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ABSTRACT 


This thesis addresses the problem of optimally re-assigning strike aircraft 
to targets in response to the emergence of “pop-ups” or time-sensitive targets. 
The first part of this thesis develops an automated decision aid to rapidly revise 
the current air tasking order (ATO), so as to: maximize achievement of target 
destruction goals (weighted by target priorities), minimize attrition risk to 
employed assets, and disrupt the current ATO as little as possible. The second 
part of the thesis develops a detailed test and evaluation plan to conduct a 
comparison of two competing automated decision aids and the current manual 
reassignment methods. Critical operational issues, measures of effectiveness 
and measures of performance were developed to fully evaluate operational 
performance. The time-sensitive-targeting decision aid was tested and validated 
during major air strike live exercises at Marine Aviation Weapons and Tactics 
Squadron One. Careful measurements comparing the re-taskings recommended 
by the decision aid against actual decisions demonstrated that in every case the 
model's solutions were of better or equal quality, maximized combat asset 


utilization, and were achieved significantly faster. 
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EXECUTIVE SUMMARY 


The emergence of Time Sensitive Targets (TST) poses an imminent threat 
to friendly forces and the successful completion of the friendly commanders 
mission. Failure to decide and act within a short period of time, specified by a 


target’s vulnerability window, results in loss of life or the disruption of a vital area. 


The intent of this thesis is to provide the tactical decision maker with a fast 
and accurate automated decision aid that has been successfully demonstrated in 
a live tactical environment. The decision aid develops an appropriate aviation 
response to the emergence of a TST. There exists a multitude of systems that 
provide a commander with the necessary information to develop feasible courses 
of action. However, the determination of the best asset to assign to each target 


is done manually, and based solely on the experience of the decision maker. 


The decision aid employs optimization techniques, and is an adaptation of 
the Davi Castro Model [2003] developed at the Naval Postgraduate School. The 
optimization model takes in all available resource information provided by the Air 
Tasking Order (ATO), to include; the number and type of aircraft, ordnance load, 
time window, and the aircrafts previously assigned task. Coupled with 
commander's guidance in the form of a prioritized list of targets, and assessment 
of the threat, the decision aid determines which asset to assign to each target 
within the vulnerability window. Options considered by the decision aid include: 
maintain the current plan, assign ground alert aircraft, divert airborne missions, or 
combine multiple mission assets against a single target. Assignments are made 
by weighting the terms of the objective function so as to consider with each 
assignment: target precedence, risk to aircrew, the number of required ATO 
changes, and available asset range. 


In order to make a determination on whether or not this decision aid can 
support the needs of the commander a tactical evaluation was conducted at 


Marine Aviation Weapons and Tactics Squadron One, Yuma, Arizona. A 


xix 


determination was made based on the results of the Critical Operational Issues 
(COl) listed below: 


COI 1. Target Selection. The decision aid properly conducts target 
prioritization of targets to ensure mission accomplishment of higher 
priority targets before lower priority targets. 

COI 2. Asset Availability. The decision aid properly recognizes the 
assets that are available for a particular target. 

COI 3. Target Asset Pairing. The decision aid properly recommends 
assets that have the ability to destroy the target. 

COI 4. Mission Risk Assessment. The decision aid accurately assesses 
the risk of its proposed assignments. 

COI5. Persistence. The decision aid minimizes the number of changes 
to the ATO to achieve mission accomplishment. 

COI 6. Timeliness. The decision aid provides a proposed solution fast 
enough to be effective when compared to current methods. 

COI 7. Options. The decision aid did not provide the decision maker with 
multiple alternative solutions for mission re-tasking when available. 

COI 8. Tactical Accuracy. The decision aids output provide tactically 
acceptable assignments. 

COI 9. Interoperability. The decision aid operates correctly with the 
current C4l architecture. 

COI 10. Software Reliability. The decision aid operates continuously 
without interruption or failure for a prolonged period of time. 


Careful measurements comparing the re-taskings recommended by the 
decision aid against actual decisions demonstrated that in every case the 
model's solutions were of better or equal quality, maximized combat asset 
utilization, and were achieved significantly faster. The results of the evaluation 
lead to the determination of that ten of the eleven requirements were adequately 
met. Although positive comments were received in regard to Human Factors, no 


formal evaluation of this critical operational issue was performed. 


The bottom line: NPS has produced a decision aid that can significantly 


improve the combat effectiveness against the emergence of a TST. 


XX 


I. INTRODUCTION 


A. PURPOSE 

This thesis has two purposes. The first is to develop an automated 
decision aid to rapidly revise the current Air Tasking Order (ATO), so as to: (i) 
maximize achievement of target destruction goals (weighted by target priorities), 
(ii) minimize attrition risk to employed assets, and (iii) disrupt the current ATO as 
litle as possible. The second purpose is to develop a detailed test and 
evaluation plan to conduct a multiple comparison of two automated decision aids, 
and the current manual reassignment methods employed by the Marine Aviation 
Command and Control System (MACCS). This test plan was executed during 
live exercises in April 2004 at Marine Air Station, Yuma, Arizona. In addition to 
the decision aid developed by the Naval Postgraduate School (NPS), the Rapid 
Asset Pairing Tool (RAPT) developed by the Space and Naval Warfare 
Command (SPAWAR) participated in the operational test. Critical operational 
issues, measures of effectiveness and measures of performance are developed 
to fully evaluate operational performance. 
B. BACKGROUND 

1. Problem Statement 

In today’s military aviation command and control centers, decision makers 
are challenged with re-tasking previously assigned attack assets in response to 
the emergence of higher priority targets or changes in the tactical situation. This 
re-tasking must be conducted in a very short period of time so as to: achieve the 
required probability of kill, minimize the risk to aircrew from surface-to-air threats, 
and limit the amount of changes to the current ATO. 

2. Time Sensitive Target (TST) Decision Making Practices 

TSTs are “those targets requiring immediate response because they pose 
(or will soon pose) a danger, or are highly lucrative, fleeting targets of 
opportunity.” [JP 3-60] Time-Sensitive Targeting is distinguished by the fact that 
it is purely reactive in nature. Predominantly, these targets are known to exist 


but are not yet located. Therefore, TSTs are not included in advanced planning 


1 


of missions, and need to be dealt with as soon as their locations are revealed. In 
contrast, the ATO is a product of a deliberate planning process. The majority of 
ATO assignments are preplanned aviation missions against known targets with 
known locations. The current ATO is the principle tool used in determining 
available resources necessary to contend with changes in the tactical scenario, 


such as the emergence of TSTs. 


Systems currently employed to support assignment of aviation assets to 
targets of this nature focus on providing the decision maker situational 
awareness and a means to share information within the tactical data network. 
These systems are designed to support the approval process of a proposed 
asset assignment to strike the TST. The determination or selection of an asset is 
largely done by manual methods based on the experience of the command and 
control officer. The following is a general overview of the issues a command and 
control officer must consider when selecting an asset or assets for reassignment 
during the targeting phase of the TST process. 


1. Available Resources. Decision makers require constant situational 
awareness to determine potential reassignment options 

2. Weapons Effectiveness. The asset chosen by the decision maker to 
attack the target must be able to meet the specified destruction criteria. 

3. Risk Assessment. A determination as to the vulnerability to each aircraft 
chosen to attack the target must be made. This is normally done in the 
form of a risk assessment based on the enemy’s air defense capabilities. 
It is important to point out that, “Specific TSTs may be such a threat to the 
force or to mission accomplishment that the JFC is willing to accept a 
higher level of risk and attack the target immediately.” [Commanders 
Handbook for Time Sensitive Targeting, 2002] 

4. Associated Risks of Employment. This consideration is open to 
interpretation. This could imply the minimization of risk to friendly ground 
forces (fratricide), disruption of the current ATO, or minimization of 
collateral damage. 

5. Method Selection. Of the options available, the decision maker must 
select the most appropriate platform considering all of the criteria 
previously mentioned. 

6. Decision. Given the method selected, this task includes the approval of 
the method of attack, and the transition of a decision into action.' 
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These techniques are developed through the education and training of our 
decision makers, who are in many cases brought in from military specialties other 
than command and control, predominantly aviators. The decision makers’ 
knowledge of platform capabilities and the threat bring a tremendous wealth of 
experience to the role. The education and training provided by institutions such 
as MAWTS-1 focus this generic aviation experience into applied real-time 
decision-making. 

C. OBJECTIVES 

There are two objectives of this thesis. The first objective is the 

development of a Time Sensitive Targeting Decision Aid that supports the needs 


of the decision maker in a real-time tactical environment. 


The second objective is the development of a comprehensive test and 
evaluation plan to capture and assess whether or not the decision aids enhance 
the decision makers ability to conduct real-time re-targeting of aviation assets. 


The alternative approaches to be evaluated include: 


a. Current manual assignment procedures based on military judgment 
and experience taught in the Weapons and Tactics Instructors (WT]1) 
Course at Marine Aviation Weapons and Tactics Squadron 1. 

b. The Rapid Asset Pairing Tool (RAPT), a genetic-algorithm based 
optimization tool under development by SPAWAR. 


c. The integer-programming based optimization tool developed by Major 
Davi Castro and Prof. Richard E. Rosenthal of the Operations 


Research Department, Naval Postgraduate School. 


D. DECISION AID SUMMARY 

Since 2001, the Space and Naval Warfare Command (SPAWAR) has 
been developing a decision aid to help reassign strike assets when high priority 
targets emerge during the execution of an Air Tasking Order. These time 
sensitive targets or “pop-up” targets, as they are known, represent a change in 
the tactical situation, which requires dynamic re-tasking of aviation assets. NPS 


started working on the same problem in 2002. Though SPAWAR and NPS use 
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different optimization modeling approaches inside their decision aids, both seek 
to assign aircraft and weapon loadouts to targets with the following objectives: 
achieve the required probability of kill, minimize the risk to aircrew from surface- 
to-air threats, and limit the amount of changes to the current Air Tasking Order 
(ATO). 


The underlying approach in SPAWAR’s decision aid is a genetic 
algorithm currently used in the Rapid Asset Pairing Tool (RAPT). [Louis,2003] 
On 3 October 2003, RAPT was demonstrated to Captain Heckert, Program 
Manager for Naval Mission Planning and Tomahawk Command and Control 
(PMA-281). The demonstration of RAPT’s ability to quickly propose options in 
reassigning aircraft to higher priority targets was received with encouragement. 
SPAWAR was given the go-ahead to pursue current development with an 
emphasis on determining whether the results of the current system could be 


considered tactically accurate. 


Assisting SPAWAR in development of a real-time decision aid for aviation 
assets is the Operations Research Department at the Naval Postgraduate 
School. Under the direction of Professor Richard Rosenthal, Major Davi Castro 
of the Brazilian Air Force developed an integer-programming based means to 
obtain a solution to the stated problem. In December of 2002, their work resulted 
in the Static and Dynamic Optimization Models used in assigning assets to a 
prioritized list of targets. For all intents and purposes, what was developed can 
be considered an ATO builder in its simplest form. Major Castro, even then, 
realized the potential for real-time asset allocation through re-optimization of an 
existing ATO. With further enhancements to his original Static Model for asset 
allocation, the ability to apply this model to real-time re-targeting is ready to be 
tested. 

E. SCOPE AND LIMITATIONS 

The Weapons Tactics Instructors course provided a means to achieve the 
objectives of this thesis, however our evaluation was not the focus of MAWTS-1. 
Emerging high priority targets are part of the exercise scenario but may be 


limited in number. Additionally, the scenario is limited to operations in a desert 
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environment. Time Sensitive Targets that can emerge in populated areas are not 
be evaluated. The plan allows for the evaluation of human factors aspects of the 
decision aids. However, the decision aids were operated primarily by the 
developers, and therefore, it was not possible to conduct an effective evaluation 
of the human factors issues. 
F. THESIS ORGANIZATION 

Chapter II provides a more detailed insight into Time Sensitive Targeting. 
Chapter Ill provides a description of the model developed by Major Castro, and 
the revisions made to adapt the model to a real-time tactical environment. 
Chapter IV discusses the development of the critical operational issues central to 
the evaluation. Chapter V provides a detailed analysis of data collected during 
the evaluation. Chapter VI provides conclusions and recommendations based on 


this analysis. 
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ll. THE TIME SENSITIVE TARGETING PROCESS 


A. INTRODUCTION 

The Time Sensitive Targeting Process consists of six phases: find, fix, 
track, target, engage, and assess. This chapter discusses each phase, with 
emphasis on Targeting, the phase in which the decision aids are employed. 
Figure 12, TST Process in Cyclical Form, provides a simple graphic outlining the 
phases of Time Sensitive Targeting and the tasks associated with each. A 
discussion of each of these tasks is the focus of this chapter and will provide the 
reader with the necessary background to understand the functioning of the 
decision aids presented in Chapter III. 


-Assess pS 

-Report results -*Focus sensors 

*Reattack . “Locate 

recommendation -Identify ; ; 
-Determine time available 


-Order engagement 
& transmit order 
*Monitor/manage 
engagement 
-Strike (attack TST) 


*Maintain track 
-Update time available 


-Determine available resources 
-Develop options to achieve desired effect 
-Weaponeer 


-Detect JP 3-60 & MTTP Step *Satisfy restrictions 
-Deconflict Additional MTTP Step *Deconflict 
-Target Area Clearance 
a) MTTP Phase -Risk assessment 
EEE -Select method 
-Decide (final approval) 





Figure 1 The TST Process in Cyclical Form. Beside each phase of 
the process the tasks that must be performed are listed. 
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1. The Find Phase 

TSTs are usually targets known to exist but have not yet been located. 
Therefore, the first task is to find them. The “find” phase, as referred to 
previously in Figure 3, receives the focus of all intelligence collection assets 
available on the battlefield. Assets used in finding TST’s include, but are not 
limited to: Special Operations Forces (SOF), Airborne Collection assets 
(JSTARS, AWACS, and Strike Aircraft), and Space Based Collection Assets. 
Those individuals who are assigned the primary or alternate role of detecting 
TST’s receive their direction from their commanders in various forms. The 
guidance and direction for TST’s may come in a format as simple as a prioritized 


list as seen in the following example. 


“The adversary is known to possess, and has the capability to employ 
nuclear WMD against the Joint Force. Accordingly the following target types are 
designated Joint TSTs: 


e Adversary activities deploying WMD from known storage areas. 


e Known or suspected TBM with WMD payload.” [Commanders 
Handbook for Time Sensitive Targeting, 2002] 


On the other hand, the tools provided to assist in the detection of TSTs 
can be very specific, such as the following notional decision matrix depicted in 
Figure 2. The manual decision aid is used by operators throughout the 
battlefield, whose responsibility it is to assist in the prosecution of Time Sensitive 
Targets. This tool helps to clearly identify targets of this nature and provide 
guidance as to the appropriate responses to them. The first column lists the 
TST’s in order of precedence. The second column describes the specific type of 
target to be found. The third column dictates who has the authority to approve 
the striking of that target. The fourth column denotes any special requirements in 
regards to striking the target. The fifth column denotes the acceptable level of 
risk for fratricide and collateral damage. The final column provides amplifying 


comments in regards to actions taken. 


Priorit TST Target Desired Approval Additonal Accpetable Re ante 
y Type Effect Authority Restrictions Risk Level ea Notes 


Immediately with 
any asset. 
Package 
recommended 
but will go 
without 


NotifyJFC 
Higher Level immediately and 
Personel or Isolate, maintain sensor 
Hi 


On-Scene 
Flight 
Leader 


Prevent 
Launch 
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required prior 
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groups meeting | Capture, or 
x Criteria kill 


track. Package 
recommended, 
threat 
dependent. 


Hazard Analysis 
Critical Weapon Blewett required 
P movement JFC MED q : 
System B Package 
or use ; 
Required 
Neurtalize 
Critical Weapon for TST Col : 
System C campaign Chief SEAD eduited 
duration 
Specific Key Convoy or 
ground Destroy JFACC military vehicles 
force/equipment approaching 
movement phase line green 
Important Neutralize 
weapon system jor : Tot Go SEAD Required 
D campaign Chief 
duration 


Figure 2 Notional TST Decision Matrix. Depicts an example of a tool 
used by operators to help identify and determine the appropriate course of 
action for each TST. 





Once the process has been initiated with an emerging target the focus of 
this phase then becomes the determination of the target as one of four possible 


classifications. These classifications are: Probable TST, Non-TST, Unknown, or 
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not a target.4 Each determination leads to one of four possible actions by the 
decision maker. Figure 3, Find Phase: Sequence of Conditions and Actions®, 
provides the sequence just described and illustrates the four possible actions. 





Find Phase 
Find Phase Input Determination Action 


+ Enter Probable TST in Collaborative Tools (i.e. JTSTM) 
en S. Pa Probable TST + Fix, Track and Target 


id ‘ - es 
: : ‘ nonTST * Target [AW Component Priorities and Procedures, or 
4 Eating rarget—*|_nenrst | * Pass to Deliberate Planning | 
be Po 
ee * Continue Find Phase 


Not a target * Discard (or enter on NSL) 

















Figure 3 Find Phase: Sequence of Conditions and Actions. Defines the 
sequence and events that occur beginning with the emergence of a target 
and concluding appropriate action given the determination of the target. 


The final output, or conclusion of this phase, is the nomination of a 
probable TST for further consideration. This in turn carries the target into the 
next phase of the TST process. 

2: The Fix Phase 

Once an emerging target is determined to be a probable TST , the process 
continues with the “fix” phase. This phase focuses on determining the precise 
location of the target. In this phase, data may be correlated from a variety of 
sources to confirm not only the location of the target, but that it is in fact a TST. 
Aside from a location determination, an additional product of this phase is the 
“vulnerability” window.6 The task associated with the vulnerability window as 
depicted previously in Figure 3, is to determine the time available to conduct the 





4 Multi-Service Tactics, Techniques , and Procedures (MTTP) for Time Sensitive Targets, FM 
3-60.1,2004 


5 Multi-Service Tactics, Techniques , and Procedures (MTTP) for Time Sensitive Targets, FM 
3-60.1,2004 


6 Multi-Service Tactics, Techniques , and Procedures (MTTP) for Time Sensitive Targets, FM 
3-60.1,2004 


10 


strike. 


For fixed targets this window will obviously be larger than those that are 


mobile. There is no standard when it comes to the size of a vulnerability window. 
However, a brief presented in 2002 by Brigadier General Jim Morehouse, USAF, 


provided an excellent example of a vulnerability window for mobile TSTs. At that 


time, mobile Time Sensitive Targets were referred to as Time Critical Targets 
(TCT) by the United States Air Force. Figure 4, Phase Allotment Time,’ below 


illustrates the vulnerability window with respect to mobile TSTs. 


In order for a 


decision maker to be effective against mobile time sensitive targets a proposed 


asset assignment must be made within 2 minutes. 








Defeating Theater TCTs 












Solid” | 
Prep 


0 Min (T) /15 Min (0)/ <10 Min (vision) 





ASSESS 













alate] 


no 
o 
® 
oO 
2 


! Processes l 


25 Mins 
Splefelels 

) Formulate attack 
| Position weapon sys | 
| Employ weapon I 
| Weapon fly-out l 


30 







Assess 





Performance Thresholds 
on TCT 30 Min Window 








Figure 4 Phase Allotment Time. 
phase within the TST Process. 
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3. The Track Phase 

Prior to entering the “track” phase, the target has been confirmed as a 
TST and the location verified. This phase will continue until engagement has 
been completed.® It is important to the decision maker that situational awareness 
is maintained, and for this reason the coordination and focusing of additional 
battlefield sensors may be required. 

4. The Target Phase 

Although each phase within the process is of equal importance, the 
“Target” Phase is more adaptable to time saving decision aids, as will be 
discussed later in Chapter Ill. This phase, as depicted in figure 1, involves nine 
tasks that the decision maker must perform in selecting the most capable assets 


to address the target. These tasks include: 
1. Determine available assets. 
2. Develop options to achieve desired effects. 
3. Weaponeer 
4. Satisfy Restrictions 
5. Deconfliction 
6. Target Area Clearance 
7. Risk Assessment 
8. Select Method 


9. Decide (final approval) 
a. Determine Available Assets 
Available resources are those assets, aviation or surface-to-surface 
fires, which can strike the TST within the designated vulnerability window. 
Aviation assets that may be considered available consist of the following: 1) 
aircraft not currently assigned to targets and assigned to an alert status, 2) 


aircraft currently assigned a target on the ATO that is of a lower priority, 3) 
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aircraft complete within current mission and have enough ordnance and fuel to 
accept a new assignment. 

b. Options That Can Achieve Desired Effects 

Of the assets available to conduct the attack on the TST, the 
decision maker must select the asset with the proper ordnance capable of 
achieving the desired effects. The list of available resources is therefore refined 
to increase the probability of success of the higher priority mission. This 
refinement process denotes the task, and develops options to achieve desired 
effects. 

Cc. Weaponeering 

For each TST, the commander provides a definition for the desired 
effects. For example, if the TST is a Multiple Rocket Launcher (MRL) Battery, 
and the desired effect is to destroy it, the commander defines what destroyed 
means. This could be stated as the number of launchers required to be rendered 
ineffective. To achieve these desired effects the decision maker conducts the 
task of weaponeering. Weaponeering is the process of determining the quantity 
of weapons, not just the weapon type, required to achieve desired effects.9 

d. Satisfy Restrictions 

In the next task, decision maker must verify any restrictions for 
attacking the target prior to engagement. Engagement restrictions vary with 
each target and are normally dependent on the precedence or value of the TST. 
These restrictions may include: 1) Collateral Damage Guidance, 2) Rules of 
Engagement, 3) Restricted Fire Areas, and 4) Fire Support Coordination 
Measures.'° Restrictions established by the commander cannot be violated 
when striking the target. 





29 Commanders Handbook for Joint Time Sensitive Targeting, 22 March 2002 
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e. Deconfliction 

Deconfliction, denotes the coordination necessary to achieve safe 
and effective execution of the mission. The deconfliction of multiple aviation 
assets within a confined airspace requires detailed integration and cross 
component coordination. 

f. Target Area Clearance 

The next task within this phase, Target Area Clearance, is simply 

a determination of whether or not the proposed asset for attack is permitted to 
strike in the location of the TST. This determination is made by the decision 
maker during the evaluation of the nominated target and is based on the decision 
makers approval authority given the geographic location of the target. 

g. Risk Assessment 

Risk Assessment encompasses all associated risks of employment 
of a targeting solution. These risks include, but are not limited to: 1) risk of 
fratricide, 2) risk to attacking forces based on the threat, 3) risk to non- 
combatants, and 4) risk of disruption to the current plan.11 All of these risks are 
weighed against the value assessed to the TST. These values vary, however; it 
is crucial to point out that “specific joint TSTs may be such a threat to the force or 
mission accomplishment that the Joint Force Commander is willing to accept a 
higher level of risk and attack the target immediately.” [Commanders Handbook 
for Joint Time Sensitive Targeting, 22 March 2002] 

h. Select Method 

The “target” phase continues with the selection of the strike asset 
by the responsible commander. It is the culmination of all previous tasks, and 
based on the vulnerability of the target will most likely have to be done in short 
period of time. 

i. Decide 

The final task within the “target” phase is the decision made by the 


commander. This culminating point in the process should take approximately 


11 Commanders Handbook for Joint Time Sensitive Targeting, 22 March 2002 
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three to five minutes in duration and represents an approved action to be taken in 
response to the TST. 

5. The Engage and Assess Phase 

The TST process concludes with the “engage” and “assess” phase. The 
engage phase is the commander’s implementation of the decision. This phase 
involves contacting the assets selected and redirecting them from the original 
assignment. Once the assets are redirected and the attack has been conducted 
the success of the mission is assessed. If the desired effects are not achieved 
the process can begin again. 


B. CONCLUSION 

The targeting phase of the TST process is of special importance to thesis. 
It is within this phase that the decision aid developed by NPS is applied. The 
discussions within this chapter provide the necessary background to understand 
the functioning of the decision aid presented in Chapter III. 
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lil. OPTIMIZATION MODEL FOR RETASKING OF AIR 
STRIKE ASSETS 


A. INTRODUCTION 

The model discussed in this chapter is a variation of the static optimization 
model previously developed by Major Davi Castro.12 The new model optimizes 
the assignment of aviation assets to a prioritized set of targets over a short time 
horizon, taking into account the presence of surface-to-air threats. The 
performance of this model has been tested during live tactical exercises. 
B. PROBLEM DESCRIPTION 

The problem addressed is the real-time re-tasking of aviation assets in 
response to a sudden change in the current tactical situation. The type of 
change for which the model is primarily intended is the emergence of a Time- 
Sensitive Target (TST) or “pop-up.” A TST is previously designated by the 
commander as a high-priority target, but its location is not known when the Air 
Tasking Order (ATO) is created. Therefore, TST’s are not included in pre- 
planned missions and need to be addressed after their locations are revealed. 


The urgency of engaging a TST requires immediate tasking within its 
vulnerability window, as described in Chapter 2. The only assets available to 
engage the TST are those currently scheduled to perform other missions or to be 
in an alert status within the vulnerability window. The problem is then to assign 
these available assets to the TST and to the targets previously scheduled for 
attack in the vulnerability window. There is not enough time to get additional 
assets ready nor to reconfigure the ordnance of the available aircraft. Assigned 
aircraft must be available in the vulnerability window and capable of destroying 


their targets to the degree required within acceptable limits of attrition risk. 


Time-critical conditions lead to a restricted set of options for assigning 
assets to targets. Each target included in this problem must be dealt with in one 


of the following ways: 


12 Optimization Models for Allocation of Air Strike Assets with Persistence, Davi Castro, 2003 
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gis 


Maintain the current plan. That is, for a non-time-sensitive target, 
use the assets already scheduled to attack it on the ATO. This 
would be the most convenient solution from a command and control 
perspective, but it may not be feasible for all targets due to the 
added demand of the TST. 


. Employ assets on alert status. This option would be the most 


convenient way to address the TST, provided the assets on alert 
are capable of attacking the TST with sufficiently high probability of 


success and low probability of attrition. 


Redirect the assets of a previously scheduled mission to a different 
target. When missions are redirected, there is not enough time to 
reorganize the aircraft assigned to the previously scheduled 
mission into separate groups. All the assets of the previously 
scheduled mission must be kept together, whether or not their 
target changes. 


. Combine the assets of two or more previously scheduled missions 


and redirect them to a different target. 


The right choice from among these options must be chosen for each 


target in less than three minutes of the emergence of a TST. Otherwise, friendly 


forces can be endangered. 


If there are not enough available assets to strike every target, there may 


be some targets left unstruck within the vulnerability window. Priorities on targets 


are used to guide which ones are most important to strike. In solving this 


problem the objectives are to: 


le 


Maximize achievement of target destruction goals (weighted by target 


priorities). 
. Minimize attrition risk to employed assets. 
Disrupt the current ATO as little as possible. 


Minimize the distance traveled on the newly assigned missions. 
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These objectives are combined into one objective function with weights to 
insure that the first objective takes precedence over the second, the second over 
the third, etc. 


The amount of target destruction achieved from the assignment of assets 
to targets depends on known probabilities of kill for each ordnance configuration 
against each target type. The attrition risk depends on known survival 
probabilities of each aircraft type against each air defense (AD) type, and if 
suppression of enemy air defense (SEAD) is used, it also depends on known 
suppression probabilities against the AD. 

C. SUPPRESSION OF ENEMY AIR DEFENSE 

1. SEAD Increases the Number of Strike Options 

In contrast to Major Davis thesis, the model developed and tested here 
explicitly considers SEAD. SEAD assets, such as the EA-6B Prowler aircraft, are 
employed to neutralize the effectiveness of the enemy’s AD threats, such as the 
SA-6 surface-to-air missile system. SEAD does not destroy these threats, 


instead it uses electronic means to render the AD’s targeting systems ineffective. 


A key idea of Davi’s modeling is that candidate air strike packages are 
generated and evaluated against possible targets prior to optimizing for 
recommended assignments. An air strike package is a collection of aircraft and 
their associated ordnance loadouts, launched from a common airbase (or 
carrier), which can be assigned to a target. The feature of enumerating the 
possible package assignments prior to the optimization allows the nonlinear 
calculation of probabilities of kill and attrition to be handled efficiently within a 
linear optimization model. It also makes it easy to enforce constraints on 


allowable assignments. [See Davi, page 20, for details. ] 


The addition of SEAD to the modeling of the problem allows for the 
admissibility of more air strike packages than previously allowed. In Davi’s work, 
the effectiveness of a strike package was computed based on what the survival 
probabilities for the aircraft in the package would be if they were engaged by the 


target's air defense threat. In some cases, the number of aircraft actually 
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reaching the target in the absence of SEAD would be diminished to the point that 
the package is ineffective and therefore not considered any further. Assigning a 
SEAD asset to accompany a package increases the survivability of the strike 
aircraft, thereby increasing the number of options that can be considered to strike 
the target. 

2. “Man-to-Man” vs. “Zone” Defense 

The employment of SEAD assumes that the SEAD asset can support only 
one mission at a time. The basketball analogy of “man-to-man defense” is used 
to describe this assumption. “Zone defense” would allow a SEAD asset to 
suppress threats for more than one mission, but this is not considered here. 
“Man-to-man” is consistent with current tactics. It is not assumed, however, that 
SEAD assets must remain with their supported assets after the strike. The 
SEAD may begin supporting another strike package before the strike aircraft in 
the previous package return to their base(s). 

3. Implications and Assumptions of SEAD Employment 

Due to the “man-to-man” assumption, a SEAD asset can be pre-defined 
as part of a strike package. The effectiveness and attrition probabilities of the 
strike packages can still be computed prior to optimization, preserving the 
computational advantages enjoyed by Davi’s approach. The formulas used for 
these probabilities are given in Section D. They assume air superiority during the 
conduct of TST strike operations. Air superiority does not imply the absence of 
threat, but it does assume that aircraft striking a TST will not be countered with 


an overwhelming number of surface-to-air missiles. 


The probability calculations assume that each threat system will fire one 
missile at each strike aircraft. This assumption is based on air superiority during 
offensive air support operations, and the existence of a degraded enemy air 
defense system. If the assumption were wrong and missiles outnumbered 


aircraft, the aircraft survival estimates would probably be optimistic. 


The probabilities of survival and suppression included in the model runs 
reported in this thesis are unclassified and were provided by SPAWAR. 
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D. OPTIMIZATION MODEL 

This section provides the formulation of the optimization model developed 
and tested in this thesis. 

1. Strike Package Eligibility 

In Major Davi’s models, any combination of aircraft and ordnance that 
pass muster with the judgment and experience of the strike planners can be 
considered for inclusion as a potentially recommended air strike package. For 
the time-critical situation considered in this thesis, there is far less freedom of 
choice. The aircraft and ordnance configurations as they appear on the current 
ATO are the only assets that can be considered. The only types of strike 


packages allowed are: 
1. Air strike packages currently on alert status. 


2. Air strike packages currently assigned to missions within the vulnerability 


window. 


3. The union of the assets in two or more air strike packages of the first two 
types. 
4. Air strike packages of the first three types with the addition of SEAD. 


There is not enough time to break up the assets in a current mission into 
more than one air strike package. Nor is there enough time to reconfigure an 
available aircraft's weapons loadout, even if it takes a reconfiguration to achieve 


maximum effectiveness against the TST. 


The time horizon for the problem treated in this thesis is too brief to allow 
using strike aircraft against more than one target. This is in contrast to the 
dynamic model in Chapter III of Major Davi’s thesis. His dynamic model explicitly 
accounts for the time required to fly multiple missions, including the time between 
missions for reloading ordnance, refueling and making other necessary 
preparations. 

2. Indices and Sets 

The following section identifies all indices and sets used in the formulation 


of the optimization model. 
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j Targets requiring assignment (including one or more TST, and the 
targets currently scheduled on the ATO during the vulnerability 
window), €.g., {SS-21, T-72, Logistics Site,...} 


m Missions currently scheduled on the ATO to either strike targets or 
stand on alert during the vulnerability window, {M1111, M1112,...} 


n Available air strike packages, as explained in Section D.1 
{N1,N2,...} 


The resources to be allocated in this optimization model are the assets 
already assigned to the missions indexed by m. Some of these assets may 
continue to fly to their intended targets, while others may be re-tasked to a 
different target. 


There are some situations in which two targets on the ATO are logically 
connected. For example, an SS-21 may be one target defended by another 
target, say, an SA-6. The destruction of the SS-21 cannot be achieved without 
destroying the SA-6. This relationship is identified in a pre-processing step and 
treated in the model by combining the two targets into one. 


The following input data define important subsets of the index sets. They 
specify which targets may be struck by which packages and which missions’ 
assets comprise which package. 


nj(n,j) = 1 if target | can be assigned to air strike package n; 0 otherwise. 
This depends on whether the package has a sufficiently high 
probability of achieving the required destruction, and whether it can 
do so inside the vulnerability window and within an acceptable level 


of attrition risk. 


nm(n,m) = 1 if air strike package n contains the assets of mission m; 0 


otherwise. 
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3: Parameters 
The data required to execute the optimization model are the penalty costs 
associated with assigning a particular strike package to a target, and the penalty 


cost for leaving it unstruck. 
pen_nogo(j) = penalty for not striking target j 


pen_attrition(n,j) = penalty for probable aircraft loss if strike package n is 
assigned to target j 


pen_change(n,j) = penalty for causing changes in assets’ target 
assignments if strike package n is assigned to target j 


pen_distance(n,j) = penalty for aircraft travel distance if strike package n 
is assigned to target j. 


The possibility exists that at the time a TST emerges, all assets may be 
assigned targets on the ATO. Although the ATO allows for aircraft to be in alert 
status without assignment to a dedicated target, this situation cannot always be 
relied upon. Even if there are aircraft in alert status, they may not have the 
necessary ordnance configuration to successfully engage the TST. In this case 
assets will be diverted to the TST, and some other target may go unstruck. The 
model seeks in this case to choose a low-priority target for non-assignment. If a 
previously assigned target becomes unassigned after re-tasking for the TST, 


then the unassigned target will be considered in a future run of the model. 


These penalties are derived from raw problem data and the mission 
planner’s judgment, as described in Section D.7. 
4. Decision Variables 
STRIKE(n,j) = 1 if air strike package n is assigned to target j, O otherwise 


NOGO(j) = 1 if no strike package is assigned to target j, 0 otherwise 


IDLE(m) = 1 if mission m is not used in any of the assigned packages, 0 


otherwise 


The STRIKE variables need to be treated as binary decision variables. 


The other two types of variables can be treated as continuous, since the 
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constraints will force them to take on binary values when STRIKE(n,j) are all 
binary. |The variable IDLE(m) accounts for the possibility that the assets 
associated with a mission will not be needed for any of the current target 
assignments. The commander may choose to put these assets into the alert 
status. 

5. Formulation 

Minimize 

[pen_attrition(n,j) + pen_change(n,j) + pen_distance(n,j)] * STRIKE(n,j) 
(nj) ni(n,)=1 


+ >/pen_nogo()) * NOGO(j) 


s.t. 


>. STRIKE(n,j) + NOGO() = 1, Vj 


n: nj(n,j)=1 


nm(n,m)*STRIKE(n,j) + IDLE(m) = 1, Vm 


(nj): nj(n,j)=1 


The first set of constraints ensures that each target is struck by at most 
one feasible, available strike package, or the target is left unstruck due to 
insufficient assets. Because the time horizon is so short, the second set of 


constraints ensures that the assets in each mission are employed at most once. 
6. Given Data from which Parameters are Derived 


Three additional indices are needed to specify the raw input data. They 
are not needed for the optimization model formulation because they are used 


only in calculations of parameters prior to the optimization. 
a Aircraft types {FA18, EAGB.,...} 
r Threat types {SA6,SA8, ...} 


Ww Weapon types currently carried by missions on the ATO, {MK83, 
MK84.,..} 
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The raw data for an instance of the problem consist of the following 


parameters: 
P_REQUIRED(j) | Commander’s required probability of success for 
target j 
SSP_KILL(w,)j) Single-shot probability of kill for weapon type w 
against target j 
NUMAC(m) Number of aircraft in mission m on the ATO 


CONFIG(m,w) Number of type w weapons loaded on each aircraft of 


mission m on the ATO 


threat(j) = r such that target j is known to be defended by a 
threat system of type r 


SSP_SUPPRESS(r, w) Single-shot probability of suppression of a threat 


of type r by a weapon of type w 


actype(m) = a such that mission m employs aircraft type a in the 
current ATO 


SSP_SURVIVAL(r,a) Single-shot probability of survival of an aircraft of 


type a when challenging a threat of type r 


SSP_KILL(w,j) Single-shot probability of kill for a weapon of type w 
against target j 
PRIORITY (j) Commander's priority for target j. (Smaller numbers 


mean higher priority) 


PREV_ASSIGN(m,j) = 1 if mission m is assigned to target j on the ATO, 
0 otherwise. LAT(m), current latitude of the strike 


aircraft on mission m 
LON(m) Current longitude of the strike aircraft on mission m. 
PRIORITY_WEIGHT, ATTRITION WEIGHT, CHANGE_WEIGHT, 
DISTANCE_WEIGHT Weighting factors for objective function terms 
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7. Derivation of Probabilities and Penalties 


Given the data provided in the previous section the following probabilities 

and penalties are derived prior to optimization. 

a. Probability of Suppression 

Probability of suppression is defined as the probability of 
successfully disabling the threat using package SEAD assets. It is calculated 
from the single shot probabilities of suppression of each SEAD weapon in the 
package. It uses an intermediate calculation of the weapon count for strike 
package n. 


NUMWEAPON(n,w) = >) nm(n,m) * NUMAC(m) * CONFIG(n,m) 


m 


P_SUPPRESS(n,j) = 1- [](1- SSP_SUPPRESS(threat(j) “™“—""PON™ ¥ nj 


WwW 


b. Probability of Attrition 

Probability of attrition is defined as the probability of losing at least 
one aircraft from a package during a strike against a target. If there is no threat 
associated with a target, the probability of attrition is zero. If there is a threat 
associated with a target, the probability is calculated as the cumulative probability 
of SEAD assets failing to suppress the threat and the probability that at least one 
aircraft in the package fails to survive the unsuppressed threat. 


P_ATTRITION(n,j) = (1 - P_SUPPRESS(n,j)) * (1 - P_SURVIVAL(n,j)) 
GC. Probability of Survival 
Probability of survival of all aircraft in a package is calculated from 
the single shot probabilities of survival for each aircraft in the package against 


the given threat. 


P_SURVIVAL(n,j)=1- [J] (1-SSP_SURVIVAL(threaty, actypegm) “™“°™ )) 


m :nm(n,m)=1 


This method assumes a threat has the ability to take a single shot 
at each aircraft in the package. This is a worst-case scenario that can be used 


when there is incomplete information about how many weapons a threat may 
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have at-the-ready or in inventory and how quickly those weapons can be 
reloaded. 

d. Probability of Kill 

Probability of kill is defined as the probability an air strike package 
can destroy a target ignoring the interference of any threats. It is calculated from 


single shot probabilities of all weapons in the package. 


P_KILL(n,j) = 1- [](1- SSP_KILL(w,j)) SMYEAPON™ ) Yn, j 


e. Probability of Success 
Probability of success is defined as the probability of destroying a 
target without loss of aircraft. It can be computed from the probability of attrition 


and probability of kill. 
P_SUCCESS(n,j) = (1 - PLATTRITION(n,j)) * P_KILL(n,j) V nj 


This computed probability is compared to the required probability of 
success for target j to decide if assigning package n to target j should become an 
allowable option in the optimization.. 

f. Penalty for Not Striking a Target 

The penalty for not striking a target is calculated by weighting a 
priority value that increases with higher priority targets. A priority exponent 
greater than 1 (usually 2) is used to accentuate the importance of top priority 


targets relative to lower priority targets. 


PEN _NOGO(j) = PRIORITY_WEIGHT * ( 1 / PRIORITY @ ) PRORTYEXPONENT 


g. Penalty for Losing an Aircraft 
The penalty for losing an aircraft in a strike is calculated by 


weighting the probability of attrition. 


PEN_ATTRITION(n,j) = ATTRITION _WEIGHT * P_ATTRITION(n,j) 
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h. Persistence Penalty 
The persistence penalties for changing targeting assignments of 
strike aircraft are calculated by weighting the number of changes or “phone calls” 
required to implement the solution. 
PEN CHANGE(n,j) = CHANGE WEIGHT * 
(1 - PREV_ASSIGN(n,j)) V n,j 


m :nm(n,m)=1 
i. Distance Penalty 
The distance penalty is calculated by weighting the nautical miles 
required for all aircraft in a package to reach a target. 


PEN_DISTANCE(n,j) = DISTANCE_WEIGHT * 
NUMAC(m)* DISTANCE(m,j) Vn, 


m:nm(n,m)=1 


Distance can be calculated as the great circle distance between the two 
locations or obtained from mapping software if there are limitations on direct 
routes. In the case of the tactical evaluation discussed in Chapter IV, distances 
were ignored because they were judged to be too small to have a realistic impact 
on decisions. 

Ee: GRAPHICAL USER INTERFACE 

With the help of the author, Mr. Anton Rowe, a research associate in the 
NPS Operations Research Department, we designed and built a spreadsheet- 
based graphical user interface (GUI) for the optimization model described above. 
The model is extremely easy to use with this GUI. 


The principle interactive displays of the GUI are illustrated in Figures 5-7. 
Figure 5 shows the Target Screen, in which each target is listed with its priority, 
air defense threat (if any) and required probability of successful attack. There is 
an indicator field that specifies which targets are included in the current run of the 
model. In the example of Figure 5, the SS-21 on the bottom row is a TST. 
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Targets 


Probability fF 
Indude Target Priority Type Threat of Success 


TGT 1 4 Logistics Site 75% 














Figure 5 Target Screen. A screen shot provided by the model interface. It 
depicts all pre-planned and designated TSTs indicated by the far left 
column. The current threat associated with each target and the required 
probability of success are indicated in far right columns. 


Figure 6 displays the Mission Screen, which represents the current ATO. 
The missions selected on this screen are the ones whose assets are to be 
allocated in the optimization. For each mission, there is a call sign, times of 
departure and return, number and type of aircraft in the mission, weapon 
configuration, and the current ATO’s target assignment. Pressing the 
“Recommend” button causes the optimization model to be generated and solved, 
with the resulting target assignments displayed in the right-most column. It 
generally takes about three seconds on a personal computer to obtain an optimal 
solution after about a minute or two of user input time. After the optimization 
model is solved, the Target Screen is updated as in Figure 7 to show the 
important results. These include the achieved probability of successful attack on 
each target and the probability of attrition for the assigned strike packages. 
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Missions 


Incdude Mission C/S 





6113 Razor 13 


x 6115 Razor15 


6103 Storm 03 
6104 Storm 04 
6131 Latch 31 
6133 Latch 33 
6135 Latch 35 
6137 Latch 37 
6175 Latch 75 
6163 Razor 63 
6165 Razor 65 





ETD 


15:51 


16:19 
16:19 
16:34 
16:34 
16:34 
17:04 
17:49 
18:07 
18:22 


ETR 


17:07 


x 6101 Storm01 16:19 19:09 


19:09 
19:09 


# Aircraft 


2 AV8B 


x 6141 Razor41 15:51 17:07 2 AV8B 
x 6143 Razor43 15:51 17:07 2 AV8B 


2 AV8B 


x 6147 Razor 47 15:51 17:07 2 AV8B 


2 EA6B 
1 EA6B 
1 EA6B 
2 F18D 
2 F18D 
2 F1i8D 
2 F18D 
2 F1i8D 
2 AV8B 
2 AV8B 


Configuration 


[4] MK83 


Assignment 


TGT SA3 





[1] POD [1] HARM —s TGT SA3 


[1] POD [1] HARM 
[1] POD [1] HARM 
[4] MKs3 

[4] MKs3 

[4] MKs3 

[4] MKs3 

[4] Mks3 

[4] MKs2 

[2] cBus3s 


TGT 2 
SLA 30 


Recommend 








Figure 6 Mission Screen. The output from the interface of the model 
contains all the original ATO information. The left-most column indicates 
the available resources that will be the input to the model and reflect the 
assets currently available during the target’s vulnerability window. In the 
top right hand corner of the display the “Recommend” Button can be seen. 
Selecting this button will run the optimization model and place the results 
in the far right column. The recommend column identifies all assignments 


to be made. 


Targets 


Probability Probability Probability 
Incude Target iori Threat ofSuccess Achieved of Attrition 





x TGT SA3 3 : SA3 85% 89% 0% 
TGT 1 4 Logistics Site 85% 
x TGT 2 a Logistics Site 85% 93% 0% 
x T6T3 6 Assembly Area 85% 91% 0% 
ZSU 2 ZSU ZSU 85% 
SA8 2 SA8 SA8 85% 
TEL = TEL 85% 
x $$21 1 $$21 85% 89% 0% 











Figure 7 Target Worksheet. The target worksheet depicts each target 
included in the input to the model. For each target a priority and desired 
probability of success is provided. The column headed by “Probability 
Achieved” indicates for each asset assigned on the mission worksheet to a 
target, the desired probability of kill has been met. 


E; Tactical Example 


The problem illustrated in Figures 5-7 is an actual example from testing 
the model in live exercises at Marine Corps Air Station Yuma. The testing is 
described in detail in Chapters IV and V. As can be seen from Figure 7, the 
recommended solution is tactically accurate in the sense that every target is 
struck with the required probability of success. In this case, the model redirected 
a surprisingly large number of missions to different targets. This is in contrast to 
the solution obtained manually by the control group in the testing. Their solution 
involved no redirection because it engaged the TST with Mission 6145, which 
was on strip launch alert. The explanation of this initially disappointing result 
from the model was that it turned out that one of the current taskings on the ATO 
did not achieve the required probability of success. In this way, the optimization 
model identified a potentially dangerous mistake in the current ATO. This was an 
unintended benefit of using the new model. The next chapter describes the test 


plan for evaluating the new model in a live tactical exercise. 
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IV. TEST AND EVALUATION 


A. INTRODUCTION 

Test and Evaluation is the “process by which a system or components are 
evaluated or compared against requirements and specifications through testing 
or experimentation. The results are evaluated to assess progress of design, 
performance, and supportability.”. [Test and Evaluation Lecture Notes, T.H. 
Hoivik, 2004] The test and evaluation of the decision aids developed by 
SPAWAR and NPS addresses the second objective of this thesis. This chapter 
explains the methodology developed and criterion used to support the evaluation 
of both the SPAWAR and NPS Decision Aids during a realistic tactical 


environment. 


One problem in evaluating a new initiative or concept is determining 
precisely what the new initiative or concept is supposed to accomplish, matching 
the appropriate measures to determine if accomplishment has occurred, and 
identifying criteria for measurement of its overall worth. This thesis will utilize the 
Dendritic methodology’? for deriving the decision aids critical operational issues, 
measures of effectiveness, measures of performance, data requirements, criteria 


for evaluation, and estimating the overall worth of these decision aid’s. 


Proper operational test designs are critical to identification and evaluation 
of factors that influence the new concepts worth. Otherwise, many so-called 
“tests” devolve into feasibility demonstrations and the resulting data and 
outcomes may not be credible or applicable for extension into real world 
situations. Unfortunately, the highly controlled and structured test methods used 
for accepting or rejecting hypothesis with a very high degree of certainty in 
confirmatory analysis do not apply to complex operational tests. Simply, there 
are too many variables to control and realism would be sacrificed. However, 
using select principles of test and experimental design in exploratory analysis, 
important insights into factors that effect or influence the worth of initiative or 


13 Fundamentals of Military Experiment Planning, Design and Analysis, by Thomas H. Hoivik, 
Naval Postgraduate School, 2002 
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concept can be identified even though statistical significance may not be 
calculated. 14 
B. DENDRITIC METHODOLOGY 

The dendritic methodology is a process for deriving a system or concept’s 
Critical Operational Issues (COI), Measures of Effectiveness (MOE), Measures of 
Performance (MOP), and Data Requirements (DR) for the purposes of 
operational test and evaluation or experimentation. Once data requirements are 
established, factors and conditions needed for appropriate evaluation can be 
identified and appropriate operational test scenarios or profiles can be generated 


for inclusion in the overall test. 


The goal of the dendritic method is to identify relevant Measures of 
Performance (MOP) needed to analyze and evaluate performance (operational 
usefulness) of a specific concept, or initiative and to answer or resolve their 
associated operational issues. Since MOPs are usually rates, ratios, percents, or 
some quantitatively or qualitatively generated number, each MOP will define its 


required data elements. 


The dendritic method starts by identifying all the functions and tasks that a 
specific decision aid is to perform. The most important decision aid functions and 
tasks will be transformed into operational issues (questions) that need to be 
evaluated (answered) during the experiment. The MOE for a decision aid 
generally relate to the capabilities desired for each function or task. Once MOEs 
have been identified, then appropriate quantitative or qualitative MOP (rates, 
ratios, percents, etc) can be developed to evaluate the effectiveness of each 
decision aid capability. It should be noted that several MOPs may be needed for 
each MOE and several MOEs are usually needed to fully encompass the 
evaluation of an experimental issue. With required and appropriate MOEs, mini- 
scenarios or run profiles, which are sets of factors and conditions, can be 
constructed to generate the required data. The test director while maintaining 


operational realism can insert these mini-scenarios or run profiles repeatedly, 


14 Fundamentals of Military Experiment Planning, Design and Analysis, by Thomas H. Hoivik, 
Naval Postgraduate School, 2002 
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and at random, to foster efficient and credible data generation and comparative 
analysis. Therefore, even though the overall operational test may be complex 
and unstructured, the insertion of mini-scenarios or run profiles with appropriate 
systematic variation of the relevant factors and conditions will allow for enhanced 
analysis and evaluation. This will foster meaningful insight into, or identification 
of, those factors that influence the mission success or failure of a concept, 
process or decision aids. 19 

C. CRITICAL OPERATIONAL ISSUES (COl) 

First and foremost, to develop an effective evaluation the critical 
operational issues of the decision aids need to be determined. The dendritic 
method was used to develop the test plan used to evaluate the decision aids. 
The following is a summary of the process used to derive the COIl’s, MOE’s, and 
MOP’s derived to evaluate the decision aids. The first step in the dendritic 
method is to determine the decision aid’s primary functions. The major 
capabilities desired for each function are then enumerated as shown in Table 1. 
These capabilities may then be subdivided into more specific capabilities if 
desired. This initial breakdown of functions and capabilities provides the basis 


for determining critical operational issues and system measures of effectiveness. 


15 Fundamentals for Military Experiment Planning, Design and Analysis, by Thomas H. 
Hoivik, Naval Postgraduate School, 2002 
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Function 


Capability 





Target Selection 


Ensure target precedence is maintained during asset assignments 





Asset Availability 


Assignments are only comprised of assets that are able to strike the asset within the 


prescribed time. 





Target-Asset Pairing 


Method of attack that is selected must be able to achieve the prescribed destruction 


criteria. 





Mission Risk Assessment 


1. Able to asses risk to multiple aircraft platforms from various surface-to-air threats 


2. Assigns SEAD support when it is required. 





Persistence 


Limits the number of disruptions to the current ATO. 





Timeliness 


Determines solution in enough time to take effective action. 





Options 


Provides multiple courses of action. 





Tactical Accuracy 


Provides solutions that are acceptable in a live tactical environment. 





Interoperability 


Can receive Air Tasking Orders from current systems employed. 








Reliability 





Perform without interruption for 24 hours. 





Table 1. Decision Aid Functions and Capabilities. 





COls, are usually the critical functions a system must perform.16 COl’s 


provide the focus and direction for our test and evaluation. COl’s are usually 


defined as questions to be resolved during the operational test. The COl’s used 
in the evaluation of our decision aids are: 


1. COI1. Target Selection. Do the decision aids properly conduct target 
prioritization of targets to ensure mission accomplishment of higher 
priority targets before lower priority targets? 

2. COI 2. Asset Availability. Do the decision aids properly recognize the 
assets that are available for a particular target? 

3. COl 3. Target Asset Pairing. Do the decision aids properly recommend 
targets that have the ability to destroy the target? 

4. COI 4. Mission Risk Assessment. Do the decision aids accurately 
assess the risk of its proposed assignments? 

5. COI5. Persistence. Do the decision aids minimize the number of 
changes to the ATO to achieve mission accomplishment? 

6. COI 6. Timeliness. Do the decision aids provide a proposed solution fast 


16 Fundamentals for Military Experiment Planning, Design and Analysis, by Thomas H. 
Hoivik, Naval Postgraduate School 2002 
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enough to be effective when compared to current methods? 

7. COI 7. Options. Do the decision aids provide the decision maker with 
multiple alternative solutions for mission re-tasking when available? 

8. COI 8. Tactical Accuracy. Do the decision aids output provide tactically 
acceptable assignments? 

9. COI 9. Interoperability. Do the decision aids operate correctly with the 
current C4l architecture? 

10.COIl 10. Software Reliability. Can the decision aids operate continuously 
without interruption or failure for a prolonged period of time? 

11.COl 11. Human Factors. Can the typical user enter inputs and 
understand the outputs in an efficient, intuitive manner with minimal 
training? 


The following sections will describe each COI in detail, and justify their 
inclusion in the evaluation. 


12 Do the decision aids properly conduct target prioritization of 
targets to ensure mission accomplishment of higher priority 
targets before lower priority targets? 


The senior commander establishes the priority of all targets on the 
battlefield and deviations from this can only occur with the consent of that 
commander. Targets are prioritized numerically beginning with one, the highest 
priority target. Given a situation where all assets are currently assigned, the 
potential for a target not being struck does exist. | Each decision aid must be 
able to accurately assign assets so that the higher priority targets are paired with 
assets. Time Sensitive Targets are the highest priority targets on the battlefield; 
successful prosecution of these targets supercede all other priorities on the 
battlefield. 


2. Do the decision aids properly recognize the assets that are 
available for a particular target? 


Available assets are those that have not been cancelled on the ATO for 
mechanical failures and those that have not been committed to their original 
target and therefore are available for possible diversion. Current asset 
availability can be determined either manually or through automated updates 
received over the tactical data network. In either case each decision aid must be 
able to recognize changes in the status of the current ATO. This will allow each 


decision aid to properly determine the set of feasible solutions. The 
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determination of available resources is a target phase task that was previously 
identified in Chapter II. 


3. Do the decision aids properly recommend targets that have 
the ability to destroy the target? 


Each asset's ordnance effects against a given target type can be 
computed by the decision aids to estimate a level of destruction or probability of 
achieving the desired effect. This calculation must be equal to or greater than 
the destruction criteria required by the commander. Assets assigned that 
achieve the desired destruction criteria will be considered a correct assignment. 
This COI supports the following Target Phase tasks of (1) develop options to 
achieve desired effects, and (2) Weaponeer. 


4. Do the decision aids accurately assess the risk of its proposed 
assignments? 


Given a threat associated with a target, each decision aid is expected to 
compute a level of risk to the assets assignments. With regards to the decision 
aid, it is not the intent to discount or discard those assignment options that 
assume a high risk. It is, however, intended to display the risk associated with 
each potential assignment to the commander in order to make an informed 
decision. Based on this risk assessment a determination by the decision aid will 
be made to determine whether or not destruction criteria can be met or if SEAD 
assets are recommended to achieve desired effects. 


5. Do the decision aids minimize the number of changes to the 
ATO to achieve mission accomplishment? 


Each decision aid is expected to select an asset for assignment that 
causes the least impact to the existing plan. In developing a decision aid to be 
employed in a tactical environment, the number of re-assignments to be made 
and the effort required to implement the re-assignment must be considered. The 
impact to the ATO is measured by the number of radio calls required to redirect 
aircraft to their new assignments. The number of radio calls reflects the amount 
of coordination required to ensure safe and effective execution of the decision. 
Any redirection of aviation assets is a disruption; too many disruptions can result 


in an unsafe and eventually ineffective execution. 
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6. Do the decision aids provide a proposed solution fast enough 
to be effective when compared to current methods? 


The timeline needed to generate a proposed solution, decide on the 
course of action, and then implement the decision is of the utmost importance. In 
Chapter II, Figure 4, a recommended timeline for decision tools to be effective in 
support of the TST process was identified. However, given a_ tactical 
environment, appropriate timeliness may be better determined by experienced 
decision makers. To resolve this COI a comparative analysis will be conducted 
to determine if there is a significant improvement in timeliness over existing 


manual methods. 


f; Do the decision aids provide the decision maker with 
multiple alternative solutions for mission re-tasking when 
available? 


There is no standard among decision makers to obtain a baseline for how 
a proposed assignment will be perceived in regards to the effectiveness, 
assessment of risk, or impact to the ATO. For this reason the number of options 
a decision aid provides allows for flexibility among decision makers. Decision 
makers may have different experiences or willingness to accept increased risk to 
achieve the desired end state. Therefore, the more options provided by the 
decision aids the more likely a proposed assignment will be accepted. 


8. Do the decision aids outputs provide tactically acceptable 
assignments? 


On a case-by-case basis the decision aids’ proposed assignments will be 
evaluated for tactical acceptance. The assessment of this COI is qualitative in 
nature and for any given situation there can be multiple solutions that are 
acceptable. Not all of them necessarily need to result in the assignment of an 
aviation asset. For this reason a comparative analysis will not be conducted for 
this COl, rather the consistency of the decision aids to produce acceptable 
assignments will be evaluated for tactical acceptance regardless of whether or 
not they reflect the same course of action selected by the decision maker. 
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9. Do the decision aids operate correctly with the current C4l 
architecture? 


Each decision aid, to be effective, has to either interact directly with 
existing decision aids that support the tactical data network or at a minimum 
recognize the data formats used by the ATO, Message Text Format (MTF). For 
each of the decision aids evaluated, the ATO provides all the information used in 
developing a proposed solution. Therefore the decision aid must be able to read 
the ATO in its existing format, otherwise access to this required information will 
not be possible. 


10. Can the decision aids operate continuously without 
interruption or failure for a prolonged period of time? 


Decision aid interruptions can occur for a variety of reasons, both external 
and internal. External factors such as power interruptions will not be credited 
against the reliability of the decision aids. However, given a consistent operating 
environment, all interruptions in the operation of the decision aids will be 
recorded and evaluated with respect to software reliability. 


11. | Can the typical user enter inputs and understand the outputs 
in an efficient, intuitive manner with minimal training? 


Given the complexity of current command and control centers, such as the 
Tactical Air Command Center (TACC), a variety of decision aids employed by 
users place a premium on training to support usability of automated decision 
aids. With this in mind, any new decision aid has to be intuitive with regards to 
user interface. Operation of the decision aid interface will be evaluated 
throughout the test. Input/Output performance will be evaluated with respect to 
training requirements and personnel capabilities. 


D. MEASURES OF EFFECTIVENESS, MEASURES OF PERFROMANCE, 
AND DATA REQUIREMENTS 


1. Introduction 

The evaluation measures are critical to the assessment of the operational 
test. These measures express the level of success to which the desired 
outcomes are achieved. Both, MOE and MOP, will allow the results of each COl 
to be quantified and analytically compared in order to present a conclusion. 


Each COI can be resolved by a single or multiple MOEs and MOPs. 
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2. Measures of Effectiveness (MOE) 

An MOE, is a measure which expresses the extent to which the system 
accomplishes or supports a mission or the task. MOE’s are designed to address 
a COI. Each functional capability of the decision aids will help in identifying and 
defining the MOEs.17 

3. Measures of Performance (MOP) 

MOP, are a quantitative or qualitative measure of the system’s capabilities 
or specific performance function. MOP’s are usually in the form of rates, ratios, 
or percents.'® Development of the MOPs using the dendritic methodology allows 
the operational test and evaluation of outcomes, either quantitatively or 
qualitatively. Quantitative MOPs will be used whenever possible. Quantitative 
measures, such as those used to evaluate the timeliness COI, will be compared 
based on a specified numeric measurement. However, due to the fact that there 
exists little formal criteria for the decision aids to be evaluated against, qualitative 
MOPs will be required for most of the performance measures. Qualitative 
measures, such as those used to evaluate the tactical accuracy COl, will be 
evaluated based on subjective measures determined by the user. There will be 
at least one MOP associated with each MOE. To meet the needs of the 
operational test MOP’s, multiple data requirements have been derived to provide 
the necessary information to make the comparisons and resolve each COl. 

4. Data Requirements 

A test plan identifies what data requirements are needed and how they will 
be collected. The data requirements are necessary to compute each of the 
MOPs. In the case of this operational test plan, data requirements are recorded 
using data collection sheets and questionnaires. These documents along with a 
complete copy of the test plan appear in Appendix 2 of this thesis. A dendritic 
analysis can show the traceable relationship between COI’s, MOE’s, MOP’s, and 





17 Fundamentals of Military Experiment Planning, Design and Analysis, by Thomas H. Hoivik, 
Naval Postgraduate Scool, 2002 


18 Fundamentals of Military Experiment Planning, Design and Analysis, by Thomas H. 
Hoivik, Naval Postgraduate School, 2002 
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data requirements. Figure 8 shows the dendritic analysis for the COI, Mission 


Risk Assessment and its relatable MOE’s, MOP’s, and Data Requirements. 











ISSUE MOE MOP DATA 
| | RQMT 





Assessment % Risk Properly Assessed 
Accuracy Properly By DA 
Assessed 
|__| Risk Properly Assessed 
Accurate SEAD 
Recommendation % Proper # SEAD Correct 
Assignments vide ments by 


# SEAD Correct 





Col4 Mission 
Risk Assessment 








Assignments by 
LM 


# SEAD 
Assignments 
Required 























Figure 8 Dendritic Method for COI, Mission Risk Assessment. 


E. DECISION AID MOE’S, MOP’S, AND DATA REQUIREMENTS 

In this section, for each COI, the appropriate MOE’s, MOP’s, and data 
requirements (DR) necessary to complete a quantitative or qualitative analysis of 
the decision aids as a whole will be discussed. 


1. Do the decision aids properly conduct target prioritization of 
targets to ensure mission accomplishment of higher priority 
targets before lower priority targets? 


MOE 1.1 - Accurate target selection. The MOE, accurate target selection, 
is accomplished, if during post decision analysis all targets of higher priority are 
assigned assets. Specifically, if all targets cannot be assigned assets, than 


those that remain unassigned are of lower priority. 





MOP 1.1.1 - Percent properly selected. The MOP will be to determine the 
proportion assigned without violating the established priorities of the targets. 


DR 1.1.1.1 — Total missions assigned. 


DR 1.1.1.2 - Number of missions not assigned with a higher priority by the 
decision aids. 


DR 1.1.1.3 — Number of missions not assigned by decision maker (DM) 
without the aid of an automated tool. 


2. Do the decision aids properly recognize the assets that are 
available for a particular target? 


MOE 2.1 - Asset availability accuracy. The MOE, accuracy of asset 
consideration, is defined by whether or not the proposed assignments 
consistently reflect those that are available for assignment. Knowledge of the 
assets that are available is dependent upon the accuracy of information available 
within the command and control centers. For this reason the assumption is 


made that the information within the TACC is both current and accurate. 


MOP 2.1.1 - Percent available considered. The MOP will be to determine 
the percent of aircraft missions proposed for assignment that are available. 


DR 2.1.1.1 - Number of aircraft available at the time target information is 


received. 


DR 2.1.1.2 - Number of aircraft assigned that were considered available 
and capable by the DM. 


DR 2.1.1.3 — Number assigned that were considered available and 
capable by the DM without the aid of an automated tool. 


3. Do the decision aids properly recommend targets that have 
the ability to destroy the target? 


MOE 3.1 - Target Allocation Accuracy. The MOE, target allocation 
accuracy, is defined by whether or not the assets proposed for assignment have 
the ability to achieve the desired affects against the given set of targets. A 
correct target asset pairing for this MOE reflects the proper calculation of 
munitions effectiveness against the given target types. This evaluation was 
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conducted on the unclassified level so these values may not reflect the most 
accurate capabilities of the weapon system. 


MOP 3.1.1 — Percent Correct Assignments. The MOP is defined as the 
percentage of correct assignments. Assignments will be considered incorrect if a 
previously established probability of destruction is not met by the assignment. 
The data requirements necessary to conduct this assessment are: 


DR 3.1.1.1- Number of assignments made by the decision aids that meet 


the destruction criteria. 


DR 3.1.1.1- Number of assignments made by the DM that meet the 


destruction criteria. 


DR 3.1.1.2- Total number of assignments proposed. 


4. Do the decision aids accurately assess the risk of its proposed 
assignments? 


MOE 4.1 — Risk assessment accuracy. The MOE is defined by whether or 
not the level of risk depicted by the decision aids is accurate for each platform 
when a surface-to-air threat is associated with the given set of targets. The 
accuracy of the assessment is reliant on the known probabilities of kill for threat 
systems against various aviation platforms. Again, the values used to perform 


this assessment are on the unclassified level. 


MOP 4.1.1 — Percent properly assessed. The percent accurately 
assessed is qualitative in nature and a success will be determined by the 
MAWTS-1 Instructor during post event analysis. The data required to perform 
the assessment of the first MOP is: 


DR 4.1.1.1 - Computed risk assessment by the decision aids. 
DR 4.1.1.2 — Risk assessment by DM. 
DR 4.1.1.3 — Total number determined to be accurate. 


MOE 4.2- Accurate SEAD recommendation based on risk assessment. 
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MOP 4.2.1 - Percent of proper SEAD recommendations. The second 
MOP, SEAD recommendation, measures whether or not SEAD is proposed in 
addition to the asset target pairing given that it is determined necessary. The 


data required to conduct this assessment is: 


DR 4.2.1- Number of SEAD assignments made when required by decision 


aids. 


DR 4.2.1 - Number of SEAD assignments made when required by the 


decision maker. 


DR 4.2.2- Total number of SEAD assignments required. 


5. Do the decision aids minimize the number of changes to the 
ATO to achieve mission accomplishment? 


MOE 5.1 - ATO Persistence. The MOE, ATO persistence, is defined as 
the level of impact the proposed assignments have on the current plan. The 
fewer the number of physical actions required by the decision maker the more 
desirable the solution. 


MOP 5.1.1 — Average number of changes. This MOP is the average 
number of changes required by each method. For each decision aid, the 


changes will be counted and compared to each other. 
DR 5.1.1.1 - Number of changes required per TST event. 


DR 5.1.1.2 — Number of TST events. 


6. Do the decision aids provide a proposed solution fast enough 
to be effective when compared to current methods? 


MOE 6.1 - Decision Speed. The MOE, decision speed, is the time 
targeting information is received in the TACC until the decision aids provides 


output or the decision maker proposes a solution. 


MOP 6.1.1 - Average time of output generation. The average time of 
target info input and decision aid output. The following are a list of the data 
requirements needed to complete the calculations. 


DR 6.1.1.1 - Time from receipt of new target information by the decision 


aids or decision maker. 
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DR 6.1.1.2 - Time of output determined by decision aids or 


recommendation on a particular course of action by the decision maker. 


re Do the decision aids provide the decision maker with 
multiple alternative solutions for mission re-tasking when 
available? 


MOE 7.1 - Multiple alternative generation. The single MOE for this issue, 
will allow us to compare each decision aid with each other and the decision 
maker to determine whether or not each method provides the decision maker 


with an acceptable number of proposed solutions. 
MOP 7.1.1 — Average number of alternatives per TST event. 
DR 7.1.1.1 - Number of alternatives recommended by the decision aids. 
DR 7.1.1.2 - Number of alternatives provided to DM. 


DR 7.1.1.3- Number of TST events. 


8. Does the decision aid output provide tactically acceptable 
assignments? 


MOE 8.1 - Probability of correct output is tactically acceptable. This MOE 
is qualitative in nature and the correctness of the proposed solution will be 
determined by MAWTS-1 instructors. 


MOP 8.1.1 — Percent output that is tactically acceptable. 
DR 8.1.1.1 — Number of acceptable assignments. 


DR 8.1.1.2- Total number of assignments made. 


9. Do the decision aids operate correctly with the current C4l 
architecture? 


MOE 9.1 - Probability of properly receive the ATO. The MOE will 
determine if the each decision aid can receive the basic information required to 
perform all necessary calculations. The inability to do this will not allow the 


decision aids to operate. 


MOP 9.1.1 - Percent of ATOs received properly. 
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DR 9.1.1.1 - Number of Air Tasking Orders properly received and 
interpreted by the decision aids. 


DR 9.1.1.2 - Number of Air Tasking Orders. 


10. Can the decision aids operate continuously without 
interruption or failure for a prolonged period of time? 


MOE 10.1 — Reliability. The MOE will help determine if the decision aids 
can perform during sustained operations without interruption. A final 
determination for this COI will not be attained during this test alone. The 


maximum length of each tactical evolution will be 8 hours. 


MOP 10.1.1 — Mean time between failure. The MOP will be calculated by 
dividing the total time of operation by the number of internal failures. The types 
of failures will also be recorded. 


DR 10.1.1.1 — Total time. 
DR 10.1.1.2 - Number of failures. 


DR 10.1.1.3 - Types of failures. 


11. Can the typical user enter inputs and understand the outputs 
in an efficient, intuitive manner with minimal training. 


MOE 11.1 — Interface usability. The MOE will be evaluated qualitatively 


with questionnaires and interviews. 
MOP 11.1.1 - Percent operators satisfied with the interfaces. 
DR 11.1.1.1 — Number satisfied with the probability of destruction display. 
DR 11.1.1.2 - Number satisfied with the risk assessment display. 


DR 11.1.1.3 — Number satisfied with the persistence measurement 
display. 


DR 11.1.1.4 — Number satisfied with the current mission number display. 
DR 11.1.1.5 — Number satisfied with the ordnance capability display. 
DR 11.1.1.6 - Number satisfied with the display regarding the previously 


assigned target, its priority, and associated threat. 
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DR 11.1.1.7- Number satisfied with the display regarding the new target 
and its priority and associated threat 


DR 11.1.1.8 — Total number of operators 
F: CRITERIA 

1. Introduction 

The operational test criteria are expressions of the operational level of 
performance of the decision aids required by the typical user personnel to 
demonstrate mission effectiveness for specific functions during each test. The 
test criteria are not pass/fail conditions, rather, they represent a baseline for use 
in the design and evaluation for the operational test. In applying the test criteria 
to technical initiatives, it should be kept in mind that ultimately performance 


should be achieved in a realistic operational environment. 19 


Without the aid of, a specific quantitative requirement, or criteria for 
comparative analysis, an effective evaluation of the decision aids is very difficult. 
The decision aids are being evaluated without any formal criteria established. 
With this in mind, evaluating the overall worth of the decision aids are limited to: 
1) comparison to manual TST methods previously discussed in chapter two, 2) a 
pairwise comparison of the decision aids developed by SPAWAR and NPS with 
respect to the COI’s, MOE’s, and MOP’s. 

2. Test Criteria 

The criterion for this operational test is first the comparative analysis 
between the results produced by the decision aids and the manual methods of 
assigning assets to TST’s in a live tactical exercise. The agency asserting the 
manual methods during this process is the Marine Tactical Air Command Center 
(TACC). During the operational test the TACC will be manned by qualified 
Marine personnel experienced in the Time Sensitive Targeting process. Each 
TST event will provide data for comparative analysis between the output of each 
of the decision aids and the manual output. 


19 Fundamentals of Military Experiment Planning, Design and Analysis, by Thomas H. Hoivik 
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The second criteria, pairwise comparison, will be a quantitative and 
qualitative analysis of the two decision aids developed by SPAWAR and NPS. 
The comparison will be conducted over each TST event to explore differences 
and advantages of each decision aid. 

G. GENERAL TEST OPERATIONS AND OPERATIONAL SCENARIO 

1. Introduction 

The development of test scenarios, segments, and trials, is the next step in 
the test planning process. Although overarching scenarios will vary from Service 
to Service and test to test, it is the segmenting and structuring of various 
segments and trials (within the scenario) to identify factor effects that are most 


important.2° 


The overall test should be concept driven. That is, it needs to reflect the 
test objectives that answer or resolve the critical operational issues for the 
decision aids. As it pertains to this evaluation, the test needs to reflect realistic 
TST events in a threat environment. Each of these events is designed to foster 


appropriate decision making in response the TSTs in a tactical environment. 


The tradeoff is between controlling scenarios where some realism is lost, 
and having total freeplay occur where factor confounding will significantly limit 
evaluation capability.21 The exercise control group, or white force, when 
executing the overall operational scenario, will provide the method of control. 
The white force can systematically vary factors and conditions within TST events 
without the TACC knowing and introducing bias. In developing these TST 
events, some factors may be held constant and others varied for each TST. This 
will allow better identification and analysis of factors that have an important 
influence on mission accomplishment, such as the presence of surface-to-air 


weapons systems. 





20 Fundamentals of Military Experiment Planning, Design and Analysis, by Thomas Hoivik, 
Naval Postgraduate School, 2002 


21 Fundamentals of Military Experiment Planning, Design and Analysis, by Thomas Hoivik, 
Naval Postgraduate School,2002 
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2. Evaluation and Scenario Description 

The evaluation was conducted during the Weapons Tactics Instructor's 
Course (WTI) 2-04. The course is executed by Marine Aviation Weapons and 
Tactics Squadron One (MAWTS-1) located at Marine Corps Air Station (MCAS) 
Yuma, Arizona. The tools and methods to be evaluated will be located in the 
Marine Tactical Air Command Center (TACC). The TACC is the senior aviation 
command and control agency for the Marine Corps and is ultimately responsible 
for the conduct of the air war, to include real time re-tasking of aviation assets. 
WTI is designed to train and evaluate the Marine Corps experienced Aviators, 
Command and Control Officers, and Aviation Ground Support Officers in the 
conduct of their assigned specialties. Once training is complete the officers and 
enlisted personnel are sent back to their units to become the resident experts for 
the unit. Within the Marine Corps aviation community, WTI has been referred to 


as a “graduate level course for tactics.” 


The execution of a WTI course presents a unique opportunity to evaluate 
not just our Marines, but the tools developed to support the conduct of their 
mission. The course is divided into two phases: Academic and Flight. The flight 
phase of the course provides the vehicle for this evaluation. 


During the flight phase the evaluation focuses on only two separate flight 
phase evolutions. The MACCS Integrated Simulated Training Exercise 
(MISTEX), and the Final Exercise (FINEX) will comprise the specific test vehicles 


for the evaluation. 


MISTEX is a static exercise consisting of the entire Marine Aviation 
Command and Control Decision aids in the field executing a realistic, low 
intensity, tactical scenario over a two-hour period. This MISTEX scenario is 
executed twice over a two-day period to ensure the MACCS is fully operational 
prior to the beginning of flight operations. 


FINEX is the culminating exercise of the WTI course. It is highly intensive 
and demonstrates all six functions of Marine Aviation. These six functions 


include Control of Aircraft and Missiles, Anti-Air Warfare, Offensive Air Support, 
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Assault Support, Air Reconnaissance, and Electronic Warfare. The functions are 
driven by a tactical scenario where command and control officers are exercising 
their delegated authority in the direction and control of aircraft. The 
determination of whether or not the decision aids can adequately support the 


decision makers in this type of operation was assessed during this evolution. 


The scenario, for the purposes of the evaluation, consists of offensive and 
defensive combat operations involving ground and air forces against a heavy 
mechanized division in a desert environment. The enemy air capabilities consist 
of two air force groups with a mixture reconnaissance, anti-air, air to ground, and 
support capabilities. In addition, a major consideration of the friendly forces was 
the enemy’s surface-to-surface, and surface to air missile threat. The phase of 
the operation in which the decision aids are evaluated primarily involved 
offensive air support and assault support operations. The general description of 
the enemy’s capabilities provide the source of targets and threats for evaluation 
of the decision aids. The enemy forces are simulated on the ground either 
through the use of static targets or electronic threat emitters. Friendly aviation 
support, command and control, and air defense forces used in the execution of 
this scenario was live. Friendly ground combat forces will be represented by, 
one USMC infantry company, and one USMC Artillery battery. There will be no 
notional/simulated friendly aircraft during the execution of the flight phase. The 
MAWTS-1 Test Plan document is include in Appendix 2 of this thesis. 

3. Data Recording 

The results, for each event, will be collected by test personnel. Test 
personnel are separate and distinct from the operational force. The data was 
compiled using questionnaires and data collection sheets as shown in the 
MAWTS-1 Test Plan, Appendix 2. The results and analysis of the operational 
test are presented in Chapter V. 

4. Operating Forces 

The operational force directly relating to the evaluation is representative of 
Marine Air Command and Control System, approximately 900 Marines. The 
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representative air wing, which supports all aviation operations within the 


scenario, provides approximately 100 daily aviation sorties. 
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V. RESULTS AND DISCUSSION 


A. INTRODUCTION 

On April 16, 2004, the test and evaluation of the decision aid concluded 
with the completion of the FINEX 3 evolution of the Weapons Tactics Instructors 
Course (WTI). During the five flight phase evolutions; MISTEX 1, MISTEX 2, 
FINEX 1, FINEX 2, and FINEX 3, twenty three Time Sensitive Targets emerged 
as part of the tactical scenario. Detailed data with respect to the COI’s discussed 
in Chapter IV were collected with the exception of COI 11, Human Factors, due 
to the high tempo of exercise operations. 


Originally, it was the intent of the evaluation to compare the results of the 
SPAWAR’s genetic algorithm. Unfortunately, due to software difficulties their 


decision aid was unavailable to be evaluated adequately. 


The following sections of this chapter provide the answers to each 
question posed by the COl’s provided in the test plan. 


B. COI 1: TARGET SELECTION. DO THE DECISION AIDS PROPERLY 
CONDUCT TARGET PRIORITIZATION OF TARGETS TO ENSURE 
MISSION ACCOMPLISHMENT OF HIGHER PRIORITY TARGETS 
BEFORE LOWER PRIORITY TARGETS? 


This COI addresses the ability of the decision aid to properly conduct 
target prioritization by assigning assets to only the highest priority targets. 
During each evolution a prioritized target list was provided. During the MISTEX 
evolutions the predominant number of ATO assignments were a pre-planned 
mission against prioritized targets. However, during the FINEX evolutions the 
majority of air missions were in general support and on alert status. A violation 
or miss-assignment as identified in Table 1 occurs if: 


1. A TST was ignored, and not assigned an asset. 

2. Of the set targets addressed during the time horizon defined by the 
TST’s vulnerability window, a higher priority target was left 
unassigned. 


Table 2, provides all data obtained with respect to this COI. 
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COI #1 Target Selection 


# of Targets left unassigned that have higher priority than 
those assigned 


Miss-Assign Total Assign Miss-Assign Total Assign 
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Table 2 COl 1, Target Selection. Depicts the data required to 
determine percentage of target selection done accurately. 


Based on the data collected the NPS Decision Aid provided accurate 
target selection 100%(34/34) of the time. By comparison the decision maker 
accurately selected the targets to be assigned assets 76 % (13/17) of the time. 


The difference in TST events between the decision aid and the decision maker is 
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the result of one of the following reasons: target location was not accurate, or the 
target was mistakenly not assigned an asset. Of the miss-assignments that 
occurred the following are cited as examples. In the first TST, the decision 
maker chose to ignore the target and execute the ATO as planned. In the 
seventh TST, occurring during MISTEX 2, the decision maker diverted a mission 
assigned to a target of priority 3, while an asset on ground alert remained 
available with the capability to destroy the target. In some cases the decision 
maker made no assignment. This action did not count for or against the decision 
maker, because it indicates an option selected other than an aviation asset. 


Within the scope of the evaluation it was determined that when given a 
prioritized list of targets the NPS Decision Aid does accurately select targets for 
assignment as required by COI 1. It is recommended that no changes be made 
to the model with respect to target selection. 


C. COI 2: ASSET AVAILABILITY. DO THE DECISION AIDS PROPERLY 
RECOGNIZE THE ASSETS THAT ARE AVAILABLE FOR A 
PARTICULAR TARGET? 


The COI addressed the decision aids ability to select assets available for 
assignment. An asset not available falls into to one of the following categories: 


1. Asset canceled on the ATO due to mechanical problems. 
2. Asset that has proceeded to target. 
3. Asset that is not under control of the component reassigning it. 


Availability is based on the accuracy of the information within the TACC. If 
the information available to the decision maker and the decision aid is inaccurate, 
neither are be held accountable for a proposal based on the available 
information. Table 3, depicts the data obtained from the proposals made by the 
decision aid and the decision maker. 
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MISTEX 1 


Finex 2 Finex 1 MISTEX 2 


Finex 3 


COI #2 Asset Availability 
# of Selceted for assignment that were unavailable 


#Unavail Total Assign # UnAvail Total Assign 
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Table 3. COl 2, Asset Availability, depicts the number of assets that 
were proposed for assignment by the test subjects that were unavailable. 


Based on the manual updates allowed by the interface the NPS decision 
was able to discern which assets were available for assignment. Assuming the 
information within the TACC was accurate the NPS decision aid proposed assets 
that were available for assignment 100% (34/34) of the time, the decision maker 
employing manual methods did so 94% (16/17) of the time. 


56 


Based on the scope of this evaluation the decision aid successfully 
answered COI 2, Asset Availability. 


D. COl 3. TARGET ASSET PAIRING. DO THE DECISION AIDS 
PROPERLY RECOGNIZE THE ASSETS THAT ARE AVAILABLE FOR 
A PARTICULAR TARGET? 


This COI addressed the ability of the assets proposed for assignment by 
the decision aid to destroy the target. In each event a determination was made 
as to whether or not the required probability of destruction for each target was 
achieved. The desired probability of destruction was provided by the MAWTS-1 
Instructors. Any asset that was assigned that could not achieve the destruction 
criteria was considered a bad assignment. The scoring of this issue was done 
qualitatively by the MAWTS-1 Instructors. The results are depicted in Table 4 
below. 
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COI #3 Target Asset Planning 
Number of Selected for assignment to primary target that meet destruction criteria 


# properly 
selected 


# properly 
selected 


Total Assign 


Total Assign 







































































Table 4 COI 3, Results. 


In resolving this COI, a determination of whether or not the decision aid 
met the established criteria was simplified because the probability of successfully 


destroying the target is provided as part of the output seen in Figure 9, below. 


58 


This figure provides an example of the results achieved during the evaluation of 
TST 5, during MISTEX 1. 





Lal = = =A = u —J TT T wv 


Missions 


Indude Mission C/S a1) ETR Aircraft Configuration Assignment Recommend 
6111 Razoril 15:51 17:07 AV8B 2] CBUS8 
6141 Razor 41 15:51 __17:07 AV8B 2) CBUS8 
6143 Razor43 15:51 17:07 AV8B [4] MK83 
6115 Razori5 15:51 17:07 AV8B [4] MK83 


6147 Razor47 15:51 17:07 AV8B [6] MK83 
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B 

t 6135 Latch35 16:34 17:54 2 F18D [4] MK83 T6T3 

B 6137 Latch37 17:04 18:24 2 F18D [4] MK83 KB AA80 
B 6175 Latch75 17:49 19:09 2 F18D [4] MK83 KB AA8O 
4 6163 Razor63 18:07 19:21 2 AV8B [4] MK82 SLA 15 
B 6165 Razor65 18:22 19:36 2 AV8B [2] CBUS8 SLA 15 
p 

p 

I 
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Figure 9 TST 5, MISTEX 1 result shows the recommended 
assignment, in the far right corner, of a section of AV8B Harriers to the 
target (SA-8). 


The proposed assignment by the decision aid of an AV-8B to strike the 


SA-8 represents a solution that achieves a probability of success of 94%. This is 
depicted in Figure 10, a screen shot of the interface for the decision aid. 
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Targets 


Probability Probability Probability 





Include iori Threat of Success Achieved _ of Attrition 

Ef 
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TGT 2 5 Logistics Site 75% 

TGT 3 6 Assembly Area 75% 

zsu 2 ZSU zsu 75% 
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Figure 10. Target Pairing Results. The decision aid determined that 
by assigning a AV-8B Harrier to SA-8 a probability of success of 94% would 
be achieved. The required probability of success was only 75%. 


This same target addressed by the decision maker with no automated 
support was resolved by assigning an EA-6B Prowler and a FA-18D with an 
ordnance configuration consisting of LAU-61. In this case, this represents an 
inadequate solution as determined by the MAWTS-1 instructors. The reason for 
the determination, LAU-61s are rockets primarily used for marking targets, and 


are too inaccurate for a high probability of success to be achieved. 


The test results indicated that the decision aid proposed a solution that 
met the required probability of destruction 100% (34/34) of the time. It should be 
noted that the decision aid was operated on an unclassified system so known 
probabilities of kill used in these calculations were also unclassified. Additionally, 
the decision aid did not take into account the implications of weather on 


ordnance selection, but this was not a factor in the evaluation. 


The decision maker's proposed assignment was also evaluated 


qualitatively. The MAWTS-1 instructors determined that the accuracy of each 
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assignment, and as a result, the decision maker achieved the desired probability 
of success 82% (14/17) of the time. 


Based on the scope of this evaluation the NPS Decision Aid met the COI 
requirements. It is recommended that in further testing classified values for 
probability of kill be used while employing the decision aid. 


E. COI 4. RISK ASSESSMENT. DO THE DECISION AIDS ACCURATELY 
ASSESS THE RISK OF PROPOSED ASSIGNMENTS? 


This addresses whether or not the decision aid properly computed the 
level of risk to aircraft given the presence of Surface-to-Air Threats. Additionally, 
based on a determination of risk, an assessment of whether or not SEAD was 
appropriately recommended was also conducted. 


Table 5, provided below, presents the results pertaining to this COI. 
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COl #4 Risk Assessment 
TST/Threat Type Probability of Attrition 
eS DM 
% Total Assign % Total Assign 
- 1 0 3 0 0 
i 2 0 1 0 1 
D 3 0 1 0 1 
= 4 0 1 0 0 
5/SA8 0 1 Y 2 
6/SA8 0 1 Y 2 
N 
x 7 3 0 1 
Lu 
PA 8 0 1 0 1 
= 9/SAG 0.5 2 Y 2 
10 1 0 1 
11 1 0 1 
F 12 0 1 0 0 
ic 13/SA6 0.6 2 Y 1 
14 0 0 0 
N 
x 15/SA8 0 1 0 
= 16/SAG 0.6 2 Y 0.2 
17/SA6 0.6 2 0 0 
18/SA6 0.6 2 0 0 
Be 19 0 1 Y 0 
® 
= 20 0 1 0 1 
21/SA6 0.6 3 0 0 
22 0 1 0 1 
23/SA6 0 1 ¥ 1 























Table 5. Risk Assessment. The table identifies for each assignment 
the probability or risk of losing an aircraft as a result of a Surface-To-Air 
Threat. 


In each case the same assessment was similar for both the decision aid 
and the unaided decision maker. The probabilities of an aircraft being destroyed 
(attrition) are based on unclassified data. In Table 6 below, a determination for 
each target for the requirement of SEAD is made. Using this determination a 
comparison is conducted to determine the rate at which SEAD was properly 


recommended. The data reveals that the decision aid recommended SEAD 
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correctly 80% (4/5) of the time. By comparison the decision maker also 
recommended SEAD 80% (4/5) of time. 


SEAD Requirement 
SEAD Required SEAD Recommended 
(MAWTS-1) 
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Table 6, SEAD Recommendation. The table indicates whether or not 
the test subjects recommended the assignment of SEAD when it was 
determined necessary by the MAWTS-1 Instructors. 


Of note is TST 23, in this case the decision maker assessed the risk of 
losing an aircraft to be 60%, based on the capabilities of an SA-6. Although 
SEAD was recommended in this case, none was available. The decision maker 
chose to assign an aircraft and accept the risk. By comparison, the decision aid 
did not recommend SEAD. Instead the proposed solution was to assign an 
available section of AH-1W helicopters. Due to their ability to use terrain to mask 
themselves from the threat, the risk of attrition was limited. What is not assumed 
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by the decision aid is that the threat, an SA-6, is customarily accompanied by a 
low altitude anti-aircraft artillery (AAA) threat, resulting in a poor assignment. 


Based on the scope of this evaluation it can be concluded that the 
decision aid accurately assessed risk and recommended SEAD when required. 
It is recommended that the decision aid should be modified to incorporate a 
realistic employment of enemy threat systems when employing helicopter assets. 


F. COl 5. PERSISTENCE. DO THE DECISION AIDS MINIMIZE THE 
NUMBER OF CHANGES TO THE ATO TO ACHIEVE MISSION 
ACCOMPLISHMENT? 


The test determined if the decision aid minimized the number of changes 
for each TST event required to obtain the desired effects on the targets being 
attacked during the vulnerability window. The impact of the decision on the 
effectiveness of other missions was not evaluated, therefore averages could be 
misleading. To make this determination the decision maker would have to 
employ each recommendation made by the decision aid. This did not fall within 
the scope of the evaluation. The following results, displayed in Table 7, provide 


a basis for determining the average number of changes for each TST event. 
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COI #5 Persistence 


Number of Changes to the ATO Required by proposed 
solution 
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Table 7. COl 5, Persistence. The table provides the number of 
required changes to be made by the decision maker for each TST event in 
order to execute the proposed solution. 


From the table we can conclude that on average the decision aid solution 
required 1.7 (31/23) changes with a maximum of three. The decision maker 
solution on average required 0.75 changes (17/23) with a maximum of two. In 
the event that the decision maker did not require a change as indicated by an (*) 
in Table 6, one of two possibilities occurred; the decision maker chose to not 


strike the TST, or elected to address the target using indirect fires. Indirect fires 
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are not an option for the decision aid. Furthermore on average for every TST 
there were 4.7 (110/23) missions available for assignment. The number of 


available missions ranged from one to eight. 


Based on these results the conclusion is that the decision aid does appear 
to minimize the number of changes per event, but requires further testing to be 
conclusive. 


G. COI 6. DECISION SPEED. DO THE DECISION AIDS PROVIDE A 
PROPOSED SOLUTION FAST ENOUGH TO BE EFFECTIVE WHEN 
COMPARED TO CURRENT METHODS? 


This COI addresses the ability of the decision aid to propose a solution in 
enough time to strike a target within a specified vulnerability window. During this 
evaluation, no vulnerability window was specified for each target. A vulnerability 
window can vary depending on the target. The default vulnerability window for 
this exercise was determined to be 20 minutes. This window was chosen 
because it provided a more stringent case to conduct the evaluation. As a result 
the decision aid was held to a tighter requirement than the decision maker. 
Table 8 provides the decision speed for each event. The scoring of time begins 
with the receipt of a target report, it concludes with the proposal of a decision by 


the decision aid and the decision maker. 
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COI #6 Timeliness 
TST Decision Time (sec) 
NPS DM 
Input Time Solution Time Decision Time 

1 60 2 - 

2 60 2 180 
= 3 120 2 60* 
nt 4 60 2 360 
Vs 5 120 2 180 
5 6 60 2 180 

1 45 2 240 
a 2 30 2 60 
os 3 120 2 2280 
in 4 45 2 720 
S 5 45 2 360 
= 1 120 8 180 
5 2 60 2 480 
= 3 50 2 180 
re 1 30 2 180 
fs 2 30 2 60 
= 3 60 2 900 

4 30 2 1200 

1 120 2 720 

2 120 2 900 
ro) 3 50 2 540 
3 4 120 2 1200 
= 5 60 2 60 

















Table 8. COl 6. Decision Speed. The table provides the time in 
seconds for the test subjects to recommend a solution to the target. 


Based on the results depicted in the table, the decision aid on average 
provided a solution in 1.7 minutes (1167sec/23), with a range of one to two 
minutes. By comparison the decision maker provided a solution, on average, in 
8.4 minutes (11160 sec/22), with a range of one minute to 38 minutes. By 


observation there appears to be a significant military difference. 


The conclusion is that the decision aid on average does provide a 
recommendation within the time required (2-3 minutes) to make an effective 


decision. Therefore, the COI has been satisfactorily met. 
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H. COI 7. OPTIONS. DO THE DECISION AIDS PROVIDE THE DECISION 
MAKER WITH MULTIPLE ALTERNATIVE SOLUTIONS FOR MISSION 
RE-TASKING WHEN AVAILABLE? 


Every decision maker will weight the effect of a proposed solution 
differently. This can be attributed to the varying background and experience of 
each decision maker. As a result the number of options available may improve 
the quality of the decision. This COI addresses the ability of the decision aid to 
provide the decision maker with different courses of action. 


The test results showed the decision aid was only able to provide the 
decision maker with one course of action. As a result, for each TST event, the 
COI was not satisfactorily met. This can be attributed to the design of the 
decision aid. It is recommended that the decision aid be modified to provide 
multiple solutions. 


I. COI 8. TACTICALLY ACCURATE. DOES THE DECISION AIDS’ 
OUTPUT PROVIDE TACTICALLY ACCEPTABLE ASSIGNMENTS? 


The focus of this COI was to evaluate if the results provided by the 
decision aid represent assignments that were tactically accurate. The 
assessment of this COI was qualitative in nature. At the conclusion of each 
evolution, the MAWTS-1 instructor reviewed the proposed assignment for each 
TST, and a decision was made. In each of the twenty-three events the instructor 
concluded that assignments proposed by the decision aid represented tactically 
acceptable solutions. The following example shows an instance where the 
decision aid solution was considered accurate, while by comparison, the decision 


maker's solution was not. 


During FINEX 3 at 2104, a FROG-7 surface-to surface missile system was 
located and the TACC was informed. Figure 11, depicts the assets available at 
the time the target emerged. 
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Figure 11. FINEX 3 ATO. The assets outlined indicate the available 
assets at the time of the emerging TST. From these available assets a 
solution can be proposed to strike the target. 


In the vicinity of the target, an SA-8 was known to exist. Based on these 
inputs the decision aid determined the best asset for assignment to be mission 
6163, a section of AV-8B Harriers, that had launched at 2145. The mission was 
previously assigned in general support of CAS missions and was available for 
assignment to a higher priority mission. By comparison the decision maker 
chose to re-assign nothing, citing the threat as the factor. During post-evolution 
analysis the MAWTS-1 instructor determined that the SA-8 did not pose a viable 
threat to the fixed wing aircraft operating at altitude. For this example it was 
concluded that the decision aid proposed the correct course of action, while the 
decision maker did not. 


Based on the scope of the evaluation it is determined that the decision aid 
meets the tactical accuracy requirements of this COI satisfactorily. 


J. COI 9. INTEROPERABILITY. DO THE DECISION AIDS OPERATE 
CORRECTLY WITH THE CURRENT C4I ARCHITECTURE? 


This COl addresses the ability of the decision aid to interoperate 
satisfactorily in the tactical environment with existing systems. During the course 
of the evaluation, five ATO were developed and transmitted to the exercise force. 
In all cases the decision aid was able to accept the ATO and perform its 
necessary tasks. The ATO parser developed to perform this function was 
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reviewed by the unit and has since been approved for incorporation into 
everyday operations. Based on these results the requirements of the COI have 
been met. However, it is recommended that for future testing that the decision 
aid be adapted to accept real time updates over the tactical data network. This 
would enhance the effectiveness of the decision aid to determine available 
resources at a given moment. This process is currently conducted manually and 
subject to error. 


K. COI 10. SOFTWARE RELIABILITY. CAN THE DECISION AIDS 
OPERATE CONTINUOUSLY WITHOUT INTERRUPTION OR FAILURE 
FOR A PROLONGED PERIOD OF TIME? 


This COI addressed the decision aids ability to function continuously 
without interruption over the course of each evolution. The length of each 
evolution ranged from two to seven hours. During each evolution the decision 
aid operated continuously without interruption. It can be concluded, based on the 
limited scope of this evaluation that the decision aid appears to meet this COl. 
For a final determination it is recommended that the decision aid be employed 
during exercises of much longer duration. 

Ly CONCLUSION 

During the evaluation ten of the eleven planned COl’s were evaluated. 

The overall results are depicted in Table 9. 
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col NPS DECISION AID RESULT 
Adequate | Inadequate Unresolved 

COl 1 X 

COI 2 X 

COI 3 X 

COI 4 X 

cols X 

COI6 X 

COI7 X 

COI 8 X 

COlg9 X 

COI 10 X 

COI 11 X 




















Table 9 COI Evaluation Results. 


Within the scope of MISTEX/FINEX evaluation all critical issues 
addressed were satisfactorily met except for COI 7, the number of options 
provided by the decision aid. COI 11, Human Factors, was not able to be 
adequately addressed by this evaluation. 

M. TEST LIMITATIONS 

The evaluation was limited by the following factors. The scenario used in 
the evaluation did not allow for targets in an urban environment. The 
environment did not incorporate adverse weather effects, which limited the 
evaluation of asset-target pairing. Specifically, low altitude threat systems could 
be countered with high altitude tactics, therefore, eliminating the requirement for 
SEAD in most cases. Finally, the duration of the exercise limited the assessment 
of software reliability. As a result it is recommended that further testing include 


operations of at least 24 hours in duration. 
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Chapter VI, provides a summary of the conclusions and recommendations 
generated by the MISTEX/FINEX evaluations. 
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VI. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS AND IMPACT 

The emergence of Time Sensitive Targets (TST) poses an imminent threat 
to friendly forces and the successful completion of the friendly commanders 
mission. Failure to decide and act within a short period of time, specified by a 
target’s vulnerability window, results in loss of life or the disruption of a vital area. 


The tactical decision aid developed in this thesis provides the decision 
maker with a fast, accurate, and appropriate aviation response to the emergence 
of a TST. The decision aid employs optimization techniques that incorporate all 
available resource information provided by the Air Tasking Order (ATO), 
including: the number and type of aircraft, ordnance load, time window, and the 
aircrafts’ previously assigned task. Coupled with commander's guidance in the 
form of a prioritized list of targets, and assessment of the threat, the decision aid 
determined which asset to assign to each target within the vulnerability window. 


The tactical decision aid was successfully demonstrated in the live tactical 
environment at Marine Aviation Weapons and Tactics Squadron One, Yuma, 
Arizona. The results of the Critical Operational Issues (COI) addressed by the 
tactical decision aid are summarized below: 


COI1. Target Selection. The decision aids properly conduct target 
prioritization of targets to ensure mission accomplishment of higher 
priority targets before lower priority targets. 

COI 2. Asset Availability. The decision aids properly recognize the 
assets that are available for a particular target. 

COI 3. Target Asset Pairing. The decision aids properly recommend 
assets that have the ability to destroy the target. 

COI 4. Mission Risk Assessment. The decision aids accurately assess 
the risk of its proposed assignments. 

COI 5. Persistence. The decision aids minimize the number of changes 
to the ATO to achieve mission accomplishment. 

COI 6. Timeliness. The decision aids provide a proposed solution fast 
enough to be effective when compared to current methods. 

COI 7. Options. The decision aids did not provide the decision maker with 
multiple alternative solutions for mission re-tasking when available. 

COI 8. Tactical Accuracy. The decision aids output provide tactically 
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acceptable assignments. 
COI 9. Interoperability. The decision aids operate correctly with the 
current C4l architecture. 
COI 10. Software Reliability. The decision aids operate continuously 
without interruption or failure for a prolonged period of time. 


The conclusion is that the decision aid can significantly improve combat 
effectiveness against the emergence of a TST. 
B. RECOMMENDATIONS 

Based on the results of the test and evaluation the following 
recommendations are provided to enhance future performance of the decision 
aid in a tactical environment. 

1. Classified Values 

The decision aid during the test and evaluation functioned on an 
unclassified level. Therefore, it is recommended that future employment of the 


decision aid employ the following classified values: 
a. Ordnance single point probability of kill. 
b. — Aircraft Survival Probabilities. 
C. EA-6B Prowler Suppression Probabilities. 


The incorporation of more accurate, classified values, increases the 
accuracy of the overall selection of the asset selected to strike a particular target. 

2. Rotary Wing Assets 

The decision aid was expanded during the test and evaluation to 
incorporate the inclusion of rotary wing aircraft into strikes packages. When 
considering the employment of rotary wing aircraft, an integrated air defense 
system has to be assumed. Survival Probabilities for rotary wing aircraft against 
medium altitude threats, such as the SA-6, need to assume the presence of low 
altitude anti-aircraft weapon systems, even though they are not normally 
specified. This is a valid tactical assumption that was not considered in the 
development of the decision aid and is required when allowing rotary assets to 


be considered in a feasible solution set. 
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3. Interoperability 

The decision aid is currently able to receive and parse an ATO from the 
current system of record, the Theater Battle Management Core System 
(TBMCS). However, it is recommended that the decision aid be further 
enhanced to receive near real time updates in regards the aircraft status. 

4. Alternative Solution Proposal 

The decision aid currently proposes only one solution option per TST 
event. It is recommended that the decision aid provide at least three to meet the 
needs of the tactical commander, as recommended by the instructors at Marine 
Aviation Weapons and Tactics Squadron One. 

5. Software Reliability 

The duration of each exercise for the test and evaluation varied from three 
to eight hours in length. Normal tactical operations run at least 24 hours in 
length. Although no software interruptions were encountered during the test and 
evaluation, it is recommended that more testing be done to support the 
conclusion of reliability. 
C. FURTHER RESEARCH 

The following sections provide recommendations for additional research 
stemming from this thesis. 

1. ATO Development 

The decision aid has proven effective in re-optimizing the current ATO 
due to an immediate change in the tactical situation. Testing revealed that 
employing an optimization tool during ATO development, as well as, execution 
could have a similar impact in solution time and accuracy of aircraft assignments. 

2. Weather Effects 

To consideration of weather effects on weapon selection is not a 
consideration of the current decision aid. The incorporation of this type of effect 
will further improve the accuracy of each asset assignment. 
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3. Collateral Damage Estimation 

A TST can emerge anywhere on the battlefield. However, significant 
concern is given to the TST that emerges in an urban environment where the 
cost of attacking a TST can manifest itself in civilian casualties. The ability to 
minimize the effects of asset selection on factors outside of the target provides a 
current and tactically relevant subject for further research. 


76 


LIST OF REFERENCES 


Brigadier General Morehouse, Director of Command and Control, DCS Air 
and Space Operations, “Time Critical Targeting Brief’, Headquarters United 
States Air Force, 2002. 


Brown, G. G., Dell, R.F., and Wood, R.K., “Optimization and Persistence,” 
Interfaces. Vol 27, Number 5, September-October 1997. 


Castro, D., “Optimization Models for Allocation of Air Strike Assets with 
Persistence”, Masters Thesis, Naval Postgraduate School, 2003. 


Commanders Handbook for Joint Time Sensitive Targeting, United States 
Joint Forces Command, Joint Warfighting Center, 22 March 2002. 


FM 3-60.1,’Multi-Service Tactics, Techniques, and Procedures (MTTP) 
for Time Sensitive Targets”, Washington, D.C., 2004. 


Hoivik, T. H., “Fundamentals of Military Experiment Planning, Design and 
Analysis,” Naval Postgraduate School, 2002. 


Hoivik, T.H., “Test and Evaluation Lecture Notes Version 5.5,” Naval 
Postgraduate School, 2003. 


Joint Publication 3-60, Joint Doctrine for Targeting, Washington D.C., 
January 2002. 


Marine Aviation Weapons and Tactics Squadron 1, “TST Brief,” November 
2003. 


McDonnell, J., Gizzi, N., and Louis, S.J., “Strike Force Asset Allocation 


using Genetic Search,” Working Paper, 2001. 


Ross, S.M., “Introduction to Probability Models, Eighth Edition,” Academic 
Press, 2003. 


Ist Marine Expeditionary Force, “| MEF Fire Support Playbook”, May 2002. 


V7 


THIS PAGE INTENTIONALLY LEFT BLANK 


78 


APPENDIX A - TACTICAL DEVELOPMENT AND EVALUATION 
(TAC D&E) TEST PLAN 





Evaluation of Decision Aids for Dynamic Re-Targeting of Aviation 


Assets 


March 2004 
by 


Maj P.R. Weaver, USMC 
Project Action Officer 
Operations Research Department, Naval Postgraduate School 


Maj C. R. Doty, USMC 
Coordinating Officer 
ADT&E Department 


Maj S. P. Mowery, USMC 
Aviation Operations Analyst 
ADT&E Department 


for 


Col R. Fox, USMC 
Marine Aviation Weapons and Tactics Squadron One 
MCAS Yuma, AZ 


Estimated Completion Date: 1 MAY 2004 
LIMITED DISTRIBUTION STATEMENT: Distribution authorized to U. S. 
Government Agencies only; Tactical Development and Evaluation. Other requests for 
this document shall be referred to Commanding Officer, MAWTS-1, Attn: ADT&E, Box 
99200, Yuma, AZ 85369-9200. 


DESTRUCTION NOTICE: Destroy by any method that will prevent disclosure of 
content or reconstruction of this document. 


79 


Tactical Development and Evaluation (TAC D&E) Test Plan 


Tactical Demonstration (TACDEMO) of Decision Aids for Dynamic Retasking of Aviation 
Assets 


Submitted by: 


P.R. Weaver 
Project Action Officer 


C. R. Doty 
Coordinating Officer 


S. P. Mowery 
Aviation Operations Analyst 


Supporting: 
M. L. Ross 
C3 Department Head 
C. Pappas III 
ADT&E Department Head 
Approved: 


R. Fox 
Commanding Officer 


80 


1.0 


INTRODUCTION 
1.1 Purpose 


This tactical evaluation will assess and compare alternative methods in 
determining real time re-assignment of aircraft to time sensitive or high priority 


targets through either automated or manual means. These will include: 


1. Current manual assignment procedures based on military judgment and 


experience taught in the Weapons and Tactics Instructors (WTI) Course. 
2. Currently Available Software Packages employed by operating forces. 


3. The Rapid Asset Pairing Tool (RAPT), a genetic-algorithm based optimization 
tool under development by SPAWAR. 


4. An integer-programming based optimization tool developed by Major Davi 
Castro and Professor Richard E. Rosenthal of the Operations Research 
Department, Naval Postgraduate School (NPS). 


1.2 System Description 

This tactical evaluation will be accomplished within the Marine Tactical Air 
Command Center (TACC) present during the Weapons Tactics Instructors (WTI) 
Course. Assignment of aircraft to reactive targets by employing manual methods 
will be conducted by the students of the Weapons and Tactics Instructor's 
Course. These students are experienced officers in the aviation community 
going through instructor qualifications. These students will be aided by the 
employment of current command and control systems that may or mat not 
employ any optimization techniques. RAPT, an optimization tool designed by 
SPAWAR, has the capability to be integrated into the exercise command and 
control architecture and receive near real time updates concerning the friendly 
and enemy situation. These updates include but are not limited to aircraft 
availability, time on station, and ordnance load out. If this level of integration is 
unable to be accomplished, RAPT as well as a similar tool developed by NPS 
can and will operate independently by relying on manual inputs by the test 
personnel. The ultimate goal of this evaluation is to determine the tactical quality 

81 


of each systems output compared to that of an experienced operator using 
current methods. A reliance on manual inputs of the system will merely identify a 
need for further development in integration of the system. 


2.0 Mission Need and Operational Requirement 
2.1 Mission Need 


The following mission need statement outlined in the Joint Mission 
Planning System (JMPS) Operational Requirements Document (ORD) has been 
the guiding element in developing the Rapid Asset Pairing Tool (RAPT) by 
SPAWAR. To achieve the Joint Vision 2020 (JV2020) operational concepts of 
dominant maneuver and precision engagement, future systems must be 
designed to streamline many areas of the targeting/decision cycle, including 
receipt of initial tasking/Commander’s Intent, gathering/disseminating 
intelligence, joint/unit-level detailed planning, dynamic mid-mission re-tasking, 
and mission effectiveness assessment. Factors driving the need for streamlined 
and shorter mission planning cycle times include mobility of targets, the 
complex/dynamic environment of large numbers of joint assets, and greater 
dependency of modern aircraft/precision weapons on large quantities of 
intelligence data. In this highly dynamic battlespace, successful execution will 
depend on a planning system that provides seamless and efficient access to all 
required data sources, including those maintained on systems operating at 
differing classification levels, and manipulates the data to provide information, 
data products and options to planners in user friendly formats. JV 2020 
envisions expanding roles for multinational and interagency partners, which will 
require collaborative planning capabilities, technological compatibility/ 


interoperability, and mechanisms for efficient information sharing. 


The focus in the development of RAPT has been on three key elements 
identified within the mission need statement. They are: “to streamline many 
areas of the targeting/decision cycle, dynamic mid-mission re-tasking, and 
mission effectiveness assessment.” [JMPS ORD, 2002] 
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2.2 


Operational Requirement 


The following is a list of user defined operational requirements to be 


used in the evaluation of the systems developed by SPAWAR and the Naval 


Postgraduate School. 


System software will be able to be mounted on current deployable computing 


hardware. 


System will be able to operate continually without interruption for 72 hours. 

System will be able to determine most effective asset to target pairing to be used 
given an aircraft type, ordnance load, and associated threat type. 

System will be able determine overall risk to assigned aircraft given a specific threat. 
System will be able to receive an ATO transmitted by TBMCS. 

Aircraft assignment will appear as a system output in a timely manner. 

System will be able to measure current impact of decision to the Air Tasking Order. 
System will be intuitive and require minimal training. 


3.0 SCOPE OF THE EVALUATION 


3.1 


Critical Technical Parameters 


The following is a list of user defined Critical Technical Parameters to be 


used in the evaluation of the systems developed by SPAWAR and the Naval 


Postgraduate School. 


At a minimum the system will be able to receive data from TBMCS and the 
Joint Mission Planning System (JMPS). 
Proposed assignment will have a probability of destruction = prescribed 
destruction criteria for the given target. 
System will determine either a probability of survival or expected attrition rate of 
assets assigned against the given target. 
System output will provide at a minimum, 3 proposed target assignments, assets 
permitting. 
Proposed aircraft/target assignments will be completed within one minute. 
System can parse an ATO with a minimum 100 sorties and maximum 5000. 
System will display at minimum: 

o Probability of Destruction 

o Risk Assessment or Expected Value of Attrition 
Persistence measurement will convey number of changes or percentage impact 
to the ATO 
Current Mission Number 
Ordnance Load-out 
Previously Assigned Target and its priority 
New Target, its Priority, and Associated threat 
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3.2 


Critical Operational Issues 
COI 1.0 - Target Selection: Does the system properly assess 


prioritization of targets to ensure mission accomplishment of higher priority 
targets before lower priority targets? Targets are prioritized numerically 
beginning with one, the highest priority target. Given a situation where all assets 
are currently assigned the potential for a target not being struck exists. Each 
decision tool must be able to accurately assign assets so that the higher priority 
targets are paired with assets. 
MOE 1.1 - Accurate target selection 
MOP 1.1.1 - Percent properly selected 
DR 1.1.1.1 — Total Missions assigned. 
DR 1.1.1.2 - Number of Missions not assigned with a higher priority by 
RAPT or NPS tool. 
DR 1.1.1.3 — Number of Missions not assigned by DM without the aid 


of an automated tool 


COI 2.0 - Asset Availability: Does the system properly determine the 
assets that are available for a particular target in its analysis? Available assets 
are those that have not been cancelled on the ATO for mechanical failures or 
those that have been sent on to original target. 

MOE 2.1 - Asset consideration accuracy 

MOP 2.1.1 - Percent available considered. Determined by the number of 
assignments made that consisted of available missions. This does not include 
missions that have already proceeded to original target, or that have been 
canceled due to maintenance. 

DR 2.1.1.1 - Actual number of aircraft available and capable 

DR 2.1.1.2 - Number assigned that were considered available and 
capable by the NPS or RAPT tool. 

DR 2.1.1.3 — Number assigned that were considered available and 


capable by the DM without the aid of an automated tool. 
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COI 3.0 - Target Asset Pairing: Does the system ensure aircraft are 
properly reassigned to targets they can destroy the target? Each mission’s 
ordnance load against a given target can be computed by the decision aids to 
determine a level of destruction. We wish to determine if this destruction level is 
equal to greater than the destruction criteria required for each target assigned an 
asset by the decision aid. Assets assigned that meet the desired destruction 
criteria will be considered a correct assignment. 

MOE 3.1 — Allocation Plan 

MOP 3.1.1 — Percent Correct Assignments 

DR 3.1.1.1- Number of assignments made by NPS or Rapt tool that 
could not destroy targets 

DR 3.1.1.1- Number of assignments made by DM without the aid of an 
automated tool that could not destroy targets 

DR 3.1.1.2- Total Number of assignments 


COI 4.0 - Mission Risk Assessment: Does the system accurately assess 
risk in its reassignments? Given a threat associated with a given target each 
decision aid is expected to compute a level of risk to the assets proposed for 
assignment. 

MOE 4.1 — Rate at which is accurately assessed 

MOP 4.1.1 - Percent properly assessed. Assessment to be done by 
qualified instructors on the MAWTS-1 Staff. 

DR 4.1.1.1 - Computed risk assessment for each target/aircraft pair 

DR 4.1.1.2 — Risk Assessment by DM. 
MOE 4.2- Proper SEAD recommendation based on Risk assessment 
MOP 4.2.1- Percent SEAD assignments made when required 

DR 4.2.1- Actual SEAD assignments made when required by RAPT 
or NPS tool. 
DR 4.2.1- Actual SEAD assignments made when required by DM. 
DR 4.2.2- Number of SEAD assignments requiring SEAD support. 
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COI 5.0 - Persistence in ATO: Does the system minimize the number of 
changes to the ATO to achieve mission accomplishment? Each decision aid is 
expected to select an asset for assignment that is affords the least impact to 
existing assignments. 

MOE 5.1 - ATO Persistence. 

MOP 5.1.1 — Number of changes to the current ATO 

DR 5.1.1.1 - Number changes required for re-tasking using RAPT or 
NPS tool for each new target input. 

DR 5.1.1.2 - Number changes required for re-tasking by DM without 
the aid of an automated tool for each new target input. 


COI 6.0 - Timeliness: Is the system fast enough to be effective compared 
to current methods? 
MOE 6.1 - Decision Speed 
MOP 6.1.1 - Average time to output recommendation to the DM 
DR 6.1.1.1 - Time from receipt of target, to solution recommended to 
DM with the NPS or RAPT tools 
DR 6.1.1.2 - Time from receipt of target, to solution recommended to 
DM without the decision aid. 


COI 7.0 - Options: Does the system provide the decision maker with 
multiple alternative solutions for mission re-tasking when available? 
MOE 7.1 - Multiple alternative generation 
MOP 7.1.1 - Number of provided to available alternatives 
DR 7.1.1.1 - Number of alternatives provided by the NPS or RAPT tool 
DR 7.1.1.2 - Number of alternatives provided to DM without the aid of 
an automated tool. 


COI 8.0 — Tactical Accuracy: Does the system output provide tactically 
acceptable assignments to ensure the ability of assets reassigned to accomplish 
the mission? The output for each assignment will be assessed to determine if 
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the proposed assignments are reasonable. This is not assessment of optimality 
but a determination of whether or not the proposed solution is one of many 
possibilities. 
MOE 8.1 - Probability of correct output is tactically acceptable 
MOP 8.1.1 — Percent output is tactically unacceptable 
DR 8.1.1.1 — Number of acceptable assignments 
DR 8.1.1.2- Number of assignments made 


COI 9.0 - Interoperability: Does the system operate correctly with the 
current C4l architecture? 
MOE 9.1 - Probability of properly receive the ATO 
MOP 9.1.1 - Percent of ATO received properly 
DR 9.1.1.1 - Number of ATOs received 
DR 9.1.1.2 - Number of ATOs sent 


COI 10.0 — Software Reliability: Can the automated tools operate 
continuously without interruption or failure for a prolonged period of time? 


MOE 10.1 — Reliability 
MOP 10.1.1 — Mean time before failure 
DR 10.1.1.1 — Time between each failure for the RAPT and NPS tool 


COI 11.0 - Human Factors: Can the typical user enter inputs and 
understand outputs in an efficient, intuitive manner with minimal training? 

MOE 11.1 - Display 

MOP 11.1.1 - Percent operators satisfied with the display. 
DR 11.1.1.1 — Number satisfied w/ Probability of destruction display 
DR 11.1.1.2 - Number satisfied w/ Risk Assessment display 
DR 11.1.1.3 — Number satisfied w/ Persistence measurement display 
DR 11.1.1.4 — Number satisfied w/ Current Mission Number display 
DR 11.1.1.5 — Number satisfied w/ Ordnance Capability display 
DR 11.1.1.6 - Number satisfied w/ display regarding the previously 


assigned target, its priority, and associated threat. 
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DR 11.1.1.7- Number satisfied w/ the display regarding the New Target 


and its priority and associated threat 
DR 11.1.1.8 — Total number of Operators 


Col 
Target 
Selection 


Table 1. Test Objective Matrix 


Test Objectives and Sub-Objectives 


To determine if system properly selects targets 
for assignment to ensure mission accomplishment of 
higher priority targets before lower priority targets. 
-Accuracy 
-System consistency with manual methods 





Asset 
Availability 


To evaluate if the system properly determines 
the assets that are available for a particular target. 
-Proportion considered 





Target Asset 
Pairing 


To assess if the system properly reassigns 
aircraft to targets they can destroy at a certain level of 
confidence. 

-Proportion correctly calculated 
-Probability of kill for assigned aircraft 





Mission Risk 
Assessment 


To determine if the system avoids undue risk in 
its reassignments. 
-Proportion properly assessed 
-SEAD Assignment recommendation 





Persistence in 
ATO 


To determine if the system minimizes the 
number of changes to the ATO to achieve mission 
accomplishment. 
-Proportion properly predicted 
-Persistence ratio 





Timeliness 


To assess if the system is fast enough to be 
effective compared to current methods. 
-Decision time 





Options 


To determine if the system provides the decision 
maker with multiple alternative solutions for mission 
retasking, when available. 

- Provided to available alternatives 





Tactical 
Accuracy 


To determine if system recommendations for 
asset pairing is tactically accurate. 





Software 
Reliability 


To determine if the automated tools can operate 
for continuously without failure. 
-Mean time before failure 





Interoperability 


To determine if the system operates correctly 
with the current C41 architecture. 
- ATOs received correctly 
- Updates received 





Human 
Factors 





To evaluate if the typical user can enter inputs 
and understand outputs in an efficient, intuitive manner 
with minimal training. 

-Displays 
-Training time 
-Interpretation errors 
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3.4 


General Test Operations, Test Vehicles, and Scenario Overview 
3.4.1 General Test Operations 


The evaluation will be conducted during the Weapons Tactics 
Instructor's Course (WTI) 2-04. The course is executed by Marine Aviation 
Weapons and Tactics Squadron One (MAWTS-1) located at Marine Corps Air 
Station (MCAS) Yuma, Arizona. The tools and methods to be evaluated will 
be located in the Marine Tactical Air Command Center (TACC). The TACC is 
the senior aviation command and control agency for the Marine Corps and is 
ultimately responsible for the conduct of the air war, to include real time re- 
tasking of aviation assets. WTI is designed to train and evaluate the Marine 
Corps experienced Aviators, Command and Control Officers, and Aviation 
Ground Support Officers in the conduct of their assigned specialties. Once 
training is complete the officers and enlisted personnel are sent back to their 
units to become the resident experts for the unit. Within the Marine Corps 
aviation community, WTI has been referred to as a graduate level course for 


tactics. 


The execution of a WTI course presents a unique opportunity to 
evaluate not just our Marines, but the tools developed to support the conduct 
of their mission. The course is divided into two phases; Academic and Flight. 
The flight phase will provide the vehicle for this evaluation. 


During the flight phase the evaluation will focus on only the final two 
separate flight phase evolutions. The MACCS Integrated Simulated Training 
Exercise (MISTEX), and the Final Exercise (FINEX) will comprise the specific 
test vehicles for the evaluation. 


MISTEX is a static exercise consisting of the entire Marine Aviation 
Command and Control System in the field executing a realistic, low intensity, 
tactical scenario over a two-hour period. This MISTEX scenario is executed 
twice over a two-day period to ensure the MACCS is fully operational prior to 


the beginning of flight operations. 
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FINEX is the culminating exercise of the WTI course. It is highly 
intensive and demonstrates all six functions of Marine Aviation. These six 
functions include Control of Aircraft and Missiles, Anti-Air Warfare, Offensive 
Air Support, Assault Support, Air Reconnaissance, and Electronic Warfare. 
Functions are driven by a tactical scenario where command and control 
officers are exercising their delegated authority in the direction and control of 
aircraft. The determination of whether or not our tools can adequately 
support the decision makers in this type of operation will be assessed during 
this evolution. 

3.4.2 Scenario Overview 

The scenario, for the purposes of the evaluation consists of offensive and 
defensive combat operations involving ground and air forces against a heavy 
mechanized division in a desert environment. The enemy air capabilities consist 
of two air force groups with a mixture reconnaissance, anti-air, air to ground, and 
support capabilities. In addition, a major consideration of the friendly forces will 
be the enemy’s surface-to-surface, and surface to air missile threat. The phase of 
the operation in which our systems will be evaluated will primarily involve 
offensive air support and assault support operations. The general description of 
the enemies capabilities provide the source of targets and threats for our 
decision aids to be evaluated against. The enemy forces will be simulated on the 
ground either through the use of static targets or electronic threat emitters. 
Friendly aviation support, command and control, and air defense forces used in 
the execution of this scenario will be live. Friendly ground combat forces are 
represented by one USMC infantry company and one USMC Artillery battery. 
There will be no notional/simulated friendly aircraft during the execution of the 
flight phase. 

3.5 Instrumentation Requirements 

Assets required that are outside of those used to conduct the WTI course 
will be provided by SPAWAR. These requirements are limited to two Laptop 
computers capable of running the software to be evaluated. 
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3.6 Limitations and Scope of Test 
The following subsections and table provides in detail, the scope and 
limitations of the test to be conducted. A static profile as indicated on the table 
denotes no actual aircraft being flown. In this instance a simulated air picture will 
be provided to the command and control agency. A flight profile denotes the use 
of live aircraft and ordnance used during the evaluation. 
3.6.1 Phase I: Pre-Evaluation 
Testing will be conducted 2-3 April 2004 to ensure equipment functionality 
and feasibility of operations. This testing will be conducted within the TACC 
located at MCAS Yuma. We will also use this phase to record and review initial 
data. 
3.6.2 Phase Il: Final Evaluation 
Testing will commence on 13 April and conclude on 16 April 2004. This 
testing will ensure interoperability and evaluate the different decision tool’s 
command and control effectiveness. Ultimately this will provide the basis for our 
comparative analysis of the two tools and the manual methods currently 
employed by the decision makers. 
Table 2. T&E Phase Summary 


Phases Evolution Profile 


Phase I: Pre- MISTEX Static 
Evaluation 
Phase Il: Final 
SAVE UIUE- Life) a} 


13-16 April 





4.0 Operational Effectiveness 
Test E-1. Target Selection. 
Objective: To assess whether or not the methods evaluated maintain 
target priorities when reassigning assets. 


Procedure: Each target assigned attack assets on the existing ATO have 
priorities. A target with the priority of one is the highest priority. Each 


immediate time sensitive or high priority target is also assigned a similar 
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priority. As each immediate target appears during the exercise the decision 
aids will be expected to assign assets based on these priorities. If the results 
reveal that a previously assigned target that is not paired with an asset after 
running the decision aids has a higher priority than those assigned to assets it 
will be counted. Operators will annotate on the effectiveness data sheet the 
number of targets that are of higher priority than those assigned assets. 


Data Analysis: Accurate target selection will be quantitative in nature. 
The analysis will be based on the number of higher priority targets that do not 
have an asset assigned per the total number of assets assigned to targets. 
Operators will annotate on the data collection sheet (Appendix B) the number 
of targets with no assets assigned that have a higher priority than the targets 
assigned assets. During each run of the decision tool the operator will also 
annotate the number total number assignments that were considered by each 
decision aid. 

Test E-2. Asset Availability. 
Objective: | To assess whether or not the methods under evaluation 


select assets for reassignment that are currently available. 


Procedure: The operator will record the number of assignments 
suggested by the decision aid that are unavailable and annotate that on the 
effectiveness data sheet (Appendix B). For example, assets that have been 
identified to have mechanical problems, or assets that have already 
proceeded to assigned targets. This effectiveness test does not take into lack 
of proper reporting by the exercise force. Regardless of the reason for the 
decision aids consideration of unavailable assets the operator will annotate 
the number of erroneous assignments and the total number of assignments. 
The remarks section on the data sheet will be used to provide reasons for the 


decision tools assignment if erroneous. 


Data Analysis: Asset Availability will be assessed quantitatively. A 
higher ratio of miss-assignments will aid in identifying a system problem in 
recognition of available assets. The reliability of this measurement will 
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depend on the timeliness and accuracy of information about current aircraft 
status. Each method will be compared to determine which has a lower rate of 
assignment of assets that are not available. A higher rate of assignment of 
assets not available will help identify a problem with either receiving 


automated updates or manually updating asset availability. 


Test E-3. Target Asset Pairing. 
Objective: To assess whether or not the methods under evaluation 
properly reassign assets to target that have the ability to meet desired 


destruction criteria provided by the component commander. 


Procedure: Each decision aid computes the probability of destruction 
against a particular target based on the assets ordnance capability. Each 
decision aid pre-calculates these values. The desired destruction criteria is 
provided by the destruction criteria matrix developed by the exercise force. 
Operators will evaluate the number of missions reassigned to the immediate 
target that have a probability of destruction that is less than the stated 
destruction criteria. The results will be recorded by either a yes the 
destruction criteria is met by the results of the decision aid or no it is not on 
the data collection sheet (Appendix B). Additionally, the destruction level 
attained by the proposed assignment well be annotated on the data collection 


sheet. 


Data Analysis: This measure is quantitative in nature. Each method either 


automated or manual will be compared to determine which achieves a lower rate 


of inadequate assignment. The total number of assignments that are below the 


probability of destruction will be annotated on the data collection sheet. 


proposed assignment that does not meet the destruction criteria will indicate 


either an error in the method for calculating weapons effectiveness against a 


given target type or lack of availability of assets necessary to destroy a given 


target. In the case of the latter we will be able to assess the ability of the 


decision aids to assign assets that can achieve the best results possible. 
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Test E-4. Mission Risk Assessment. 

Objective: To determine whether or not each method evaluated accurately 
assesses and takes action to mitigate the risk from associated surface to air 
threats. 


Procedure: The proper assessment of risk will be determined post exercise 
by evaluating each assignment to determine an accurate risk assessment. 
Operators will maintain a record of each assignment’s risk assessment whether 
that is in the form of a probability aircraft loss or expected value of attrition. 
Based on this assessment operators will determine if the system or decision 
maker assigned SEAD assets when available to minimize risk. Operators will 
annotate risk assessment on the effectiveness data sheet (Appendix B) provided. 
The assignment of SEAD will be annotated with either a yes or a no on the data 
collection sheet (Appendix B). 


Data Analysis: Two elements will be analyzed. First, the number of proper 
risk assessments for each aircraft will be counted to determine a percentage of 
proper assessment. Second, a comparative analysis will be conducted to 
determine which method produced the assignments with minimal risk and 
assigned SEAD when required. A failure to properly assess risk to aircrew given 
a surface-to-air threat could point to a problem in the values used in the 
probability of kill foreach — threat or a problem in the formulation of risk. 

Test E-5. Persistence. 
Objective: To evaluate whether each method attempts to minimize the 


impact of the reassignment to the current ATO. 


Procedure: Each ATO can be considered a previously optimized assignment 
plan. A immediate target necessitates the re-optimization of the ATO. Each 
method when conducting the reassignment takes steps to insure that as much of 
the current ATO as possible stays intact while assigning an asset to the 
immediate target. The number of changes the ATO can be reflected in the 
number of calls that have to be made to affect the new assignment. 
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Data Analysis: A comparative analysis will be conducted to determine which 
of the methods effectively minimize the number of changes to the current ATO 
after conducting the reassignment of assets. Time to act on the proposals 
provided by the decision aids determines the usefulness of the proposal. Too 
many changes in a short period of time will indicate a solution, although 
technically superior, that may not be feasible. 

Test E-6. Timeliness. 

Objective: To determine whether or not the methods under evaluation can 

produce courses of action for the decision maker in a timely enough manner to 


assist in making a decision. 


Procedure: Throughout the conduct of the exercise immediate targets will 
appear that require assignment of assets. At each appearance the operator will 
annotate on the data collection sheet the time. Target information will be entered 
into the appropriate system to await results. Once the results are displayed the 
time will then be annotated on the data collection sheets (Appendix B). From this 
information we will determine the total time to process the information and make 
a recommendation to the decision maker. We will then determine the average 


time for each method. 


Data Analysis: The analysis is quantitative in nature. Each methods time 
will be compared to determine the method that can produce results in the 
shortest period of time. Each failure will be categorized into one of three types: 
software, hardware, or network failure and annotated on the reliability data 
collection sheet. (Appendix C) 

Test E-7. Options. 
Objective: To evaluate whether the methods provide the decision maker 


with more than one course of action. 


Procedure: For each implementation of the decision tools the number of 
options provided by the decision maker will be counted by the operator and 
annotated on the data sheet (Appendix B). For every immediate target there 


exists more than one possible solution. 
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Data Analysis: The average number of options provided by the decision 
aids will be calculated and compared between each of the methods. 
Test E-8. Tactical Accuracy. 

Objective: To evaluate whether or not the output by the automated 
decision tools provide results that can be considered tactically acceptable. This 
is one of the primary purposes of the evaluation. 


Procedure: MAWTS-1 Instructors present during the evaluation will examine 
each data sheet. 


The information on the data sheets represents the proposed solution for each 
immediate target that presents itself during the exercise. Their comments will be 


annotated on the effectiveness data sheet provided (Appendix B). 


Data Analysis: This analysis is subjective in nature. An overall assessment 
will be made as to the accuracy of the proposed assignments for each individual 
method. 


Operational Suitability 


Test S-1. Software Reliability. 
Objective: To determine if the automated tools can operate continuously 


without interruption throughout the entire tactical evolution. 


Procedure: — The tactical evolutions will run approximately eight hours. 
During this time the number of system failures will be annotated on the reliability 
data sheet provided. Each failure will be noted with the time of failure and time of 
restart. 


Data Analysis: The analysis is both objective and subjective. Quantitatively 
we will assess the mean time before failure. Subjectively we will determine if it is 
a fault of the hardware, software or exercise architecture. 

Test S-2. Interoperability. 
Objective: To determine if each system under evaluation is able to 


electronically receive the ATO. 
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Procedure: Prior to the start of each exercise an ATO will be disseminated 
by MAWTS-1. The ability to receive the ATO will be identified with either a 
yes or no on the data sheet. 


Data Analysis: The assessment will be quantitative in nature. We will 
assess which method on average is able to receive the ATO. Interoperability 
problems will be recorded as they occur. 


Test S-3. Human Factors. 
Objective: Can the display be easily interpreted with respect to its inputs and 


outputs? 


Procedure: Using the suitability questionnaire a perception of how well the 
interface display for each system explains the results to the user. Each aspect 
identified in the data requirements will be assessed. 


Data Analysis: The analysis is subjective in nature. Each method will be 


assessed individually to determine usability. 


97 


LOGISTICS AND SUPPORT REQUIREMENTS 
1.0 MAWTS-1 PERSONNEL REQUIREMENTS 
1.1. Aviation Development, Tactics, and Evaluation (ADT&E) Department 
1.1.1 Coordinating Officer 


Coordinate the efforts of all T&E participants throughout the duration of 
the T&E process. 


Co-develop T&E plan. 
Co-develop operator checklists and questionnaires. 
Provide assistance in developing the T&E final report. 
1,1,2 Aviation Operations Analyst 
Provide analytical assistance to support the T&E. 
Provide assistance to the Project Action Officer in writing the final report. 


1.2 Command, Control, and Communications (C3) Department 
1.2.1 Project Action Officer 


Coordinate the efforts of all T&E participants throughout the duration of 
the QA process. 


Co-develop the T&E plan. 
Co-develop aircrew and operator checklists and questionnaires. 
Supervise administering and collection of surveys. 
Co-develop T&E final report. 
Table 2 details project milestones. 


Table 3. Project Milestones 


Milestone Date Responsible Agency 
T&E Phase | 2-3 April 04 NPS 


T&E Phase II 13-16 April 04 








Final Report 1 May 04 MAWTS-1 


2.2 Personnel Assignment 
Personnel assigned to this T&E are listed in Table 3. 
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Table 4. T&E Personnel 


Fax maate)atss 


atclalie Name Billet E-mail (DSN) (DSN) 





Maj Weaver, P.R. Project Action Officer rweaver@nps.navy.mil 756-2786 


Doty, C.R. Coordinating Officer dotycr@mawts1.usmc.mil 269-2637 | 269-3681 
Mowery, S. P. Aviation Ops Analyst mowerysp@mawts1.usmc.mil 269-2637 269-2684 





3.0 Reports 
The results of this T&E will not be released until the final report is signed by the MAWTS-1 
Commanding Officer. 
4.0 Project Security 
This T&E is UNCLASSIFIED FOUO. 
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APPENDIX B: POST EVENT QUESTIONNAIRE 


UNITED STATES MARINE CORPS 
MARINE AVIATION WEAPONS AND TACTICS SQUADRON ONE 
BOX 99200 
YUMA, AZ 85369-9200 





TACTICAL EVALUATION OF OPTIMIZATION METHODS USED IN NEAR REAL TIME 
RETARGETING OF AVIATION ASSETS 


March 2004 


101 








Annex B: Questionnaires and Data Sheets (TO be filled out by participant 


Background. This is a Test and Evaluation (T&E) for the use of the Rapid Asset Pairing Tool 
(RAPT) and NPS Tool. 


Instructions. This survey is divided into two sections that will evaluate RAPT: 
(1) Effectiveness 


(2) Suitability 


|. Background Information 
Age: Rank (i.e. O-3): Service (i.e. USMC): Years of Service: 


Designations Held: 
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TST# 


Questionnaire 


Date 
Evolution 
Tool Assessed: RAPT NPS (Circle One) 
Target Type Target Priority 
COl #6 Timeliness 
Time Received Time Output received (automated tool) 
Time DM determined option without automated tool Time DM assigns Asset 


Remarks 


COl#1 Target Selection 

Number of targets not assigned assets that are a higher priority than those assigned 
assets. 

NPS/RAPT DM Total Number of assets 
assigned 


Remarks 


COl#2 Asset Availability 

Number of assets assigned that are available based on current situation. 
RAPT/NPS DM 

Total assets assigned 


Remarks 
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COl#3 Target Asset Pairing 
Number of assets assigned that meet destruction criteria. 


RAPT/NPS DM % Assigned by NPS/RAPT. 
Total assets assigned % Assigned by DM 


COl#4 Mission Risk Assessment 
Risk assessment to aircraft reassigned 


NPS/RAPT Prob of losing aircraft/expected value of attrition 


SEAD required YES NO 

SEAD recommended by NPS/RAPT YES NO 
SEAD used byDM YES NO 

Remarks 


COl#5 Persistence 

Number of changes to ATO using RAPT/NPS 

Number of changes to ATO required by DM actions 

Number of missions cancelled based on reassignment by RAPT/NPS 
Number of missions cancelled based on reassignment by RAPT/NPS 
Remarks 


COl#7 Options 
Number of reassignment options provided NPS/RAPT 


Number of reassignment options provided by DM 


Remarks 


COl#8 Tactical Accuracy 
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WAS THE RECOMMENDED ASSIGNMENT TACTICALLY ACCEPTABLE? YESNO 
Remarks 
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APPENDIX C: POST EVENT QUESTIONNAIRE 


UNITED STATES MARINE CORPS 
MARINE AVIATION WEAPONS AND TACTICS SQUADRON ONE 
BOX 99200 
YUMA, AZ 85369-9200 





DECISION AIDS FOR DYNAMIC RETASKING OF AVIATION ASSETS 
POST EVENT QUESTIONNAIRE 


Maj P.R. Weaver, USMC 
Operations Research Department, Naval Postgraduate School 
DSN 269-2957 COM (928) 269-2957 


Maj S. P. Mowery, USMC 


ADT&E Department 
DSN 269-2684 COM (928) 269-2684 


DECEMBER 2003 
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Background. Marine Aviation Weapons and Tactics Squadron One (MAWTS-1) is 
conducting a Tactical Demonstration (TACDEMO) of Decision Aids for Dynamic Re-tasking of 
Aviation Assets. To be filled out by participant. 


Instructions. This survey is divided into two sections that will assess the effectiveness of 
various decision aids in support of aviation assets conducting time sensitive targeting: 


(3) Operability/Reliability 


(4) Human Factors 


ll. Background Information 
Age: Rank (i.e. O-3): Service (i.e. USMC): Years of Service: 


Designations Held: 
(SAC, SWO, SWO, DBC) 


Number of WTI operations participated in: 
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Operability/Reliability. Using the scale below, rate each statement by 
circling one corresponding number. Please include comments if you have 
suggestions for improvements. Based on your experience, describe your level 
of agreement with the statement that: 


1.) RAPT/NPS was able 1 2 3 4°5: 6 7 
to receive the ATO correctly from TBMCS. 
Comments: 


2.) RAPT/NPS functioned 1 2 3 4 5 
without interruption 
Time interruption occurred Time interruption ended 


Number of interruptions 
Comments: 


General comments on RAPT effectiveness in increasing command, 
control and situational awareness. 
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Reliability Data Sheet (To be filled out by system operator for each error) 


Failure Number 

Time Failure Occurred 

Time System Repaired - 

Type of Failure (Circle One) Software Hardware Network 


Reason and Impact of Failure 
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To be filled out by participant: (Circle One) RAPT NPS Decision 
Aid 


Human Factors. Using the scale below, rate each statement by circling one 
corresponding number. Please include comments if you have suggestions for 
improvements. The following questions have to do solely with the ease of 
understanding the display information for the RAPT/NPS tools. Based on your 


experience, describe your level of agreement with the statement that: 


STRONGLY AGREE 


1.) Probability of 


destruction was easy to interpret. 


Comments: 


2.) Risk Assessment was 
easy to interpret. 
Comments: 


3.) Persistence was easy 
to interpret. 
Comments: 


4.) Current Mission number 
was displayed accurately. 
Comments: 


5.) Target information was 
easy to interpret. 
Comments: 


6.) Target priority was 
easy to interpret. 
Comments: 


6.) Associated threat 


information was easy to interpret. 


Comments: 


12 


STRONGLY 


4 


ISAGREE 
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General comments on the ease of interpreting the current display 
information: 
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